n8n-io/n8n · error · Error

bucketFromEvaluation: no fileSlug for test case "${caseDispl

Error message

bucketFromEvaluation: no fileSlug for test case "${caseDisplayPrompt(tc.testCase, tc.runs[0]?.transcript).slice(0, 60)}"

What it means

bucketFromEvaluation joins an evaluation result to the on-disk case files via a testCase→fileSlug map built from testCasesWithFiles. If an evaluation.testCases entry has no matching testCase object in that map, the comparison cannot attribute outcomes to a slug, so it throws. The message surfaces a truncated prompt so you can identify the orphan case.

Source

Thrown at packages/@n8n/instance-ai/evaluations/comparison/bucket-from-evaluation.ts:40

 * Looks up `fileSlug` by test case reference rather than array index — the
 * comparison key depends on getting the right slug, and zipping by index
 * silently miscompares if anything ever reorders the aggregate.
 */
export function bucketFromEvaluation(
	evaluation: MultiRunEvaluation,
	testCasesWithFiles: WorkflowTestCaseWithFile[],
	experimentName: string,
): ExperimentBucket {
	const slugByTestCase = new Map(
		testCasesWithFiles.map(({ testCase, fileSlug }) => [testCase, fileSlug]),
	);
	const evaluationUnits = new Map<string, EvaluationUnitCounts>();
	const failureCategoryTotals: Record<string, number> = {};
	let trialTotal = 0;
	for (const tc of evaluation.testCases) {
		const fileSlug = slugByTestCase.get(tc.testCase);
		if (!fileSlug) {
			throw new Error(
				`bucketFromEvaluation: no fileSlug for test case "${caseDisplayPrompt(tc.testCase, tc.runs[0]?.transcript).slice(0, 60)}"`,
			);
		}
		for (const sa of tc.executionScenarios) {
			const failureCategories: Record<string, number> = {};
			for (const sr of sa.runs) {
				// Verifier-incomplete runs carry no verdict — not a trial.
				if (sr.incomplete) continue;
				trialTotal++;
				if (!sr.success && sr.failureCategory) {
					failureCategories[sr.failureCategory] = (failureCategories[sr.failureCategory] ?? 0) + 1;
					failureCategoryTotals[sr.failureCategory] =
						(failureCategoryTotals[sr.failureCategory] ?? 0) + 1;
				}
			}
			evaluationUnits.set(scenarioUnitKey(fileSlug, sa.scenario.name), {
				kind: 'scenario',
				testCaseFile: fileSlug,

View on GitHub (pinned to 5ac6606e81)

Solutions

1. Pass the full set of loaded cases (no filter) to bucketFromEvaluation so every evaluated case has a slug.
2. Re-run the evaluation if cases were renamed, so testCase identities match the current files.
3. Filter the evaluation's testCases down before calling bucketFromEvaluation so orphans are dropped intentionally.

Example fix

// before: const bucket = bucketFromEvaluation(eval, filteredCases, name);
// after:  const aligned = eval.testCases.filter(tc => loadedSet.has(tc.testCase));
//        const bucket = bucketFromEvaluation({ ...eval, testCases: aligned }, loadedCases, name);

const slugByCase = new Map(testCasesWithFiles.map(({ testCase, fileSlug }) => [testCase, fileSlug]));
const orphans = evaluation.testCases.filter(tc => !slugByCase.has(tc.testCase));
if (orphans.length) throw new Error(`evaluation has ${orphans.length} case(s) absent from testCasesWithFiles`);

const isWorkflowTestCaseWithFile = (v: unknown): v is { testCase: unknown; fileSlug: string } =>
  typeof v === 'object' && v !== null && typeof (v as any).fileSlug === 'string' && 'testCase' in (v as object);

try { return bucketFromEvaluation(eval, cases, name); }
catch (e) {
  if (e instanceof Error && e.message.startsWith('bucketFromEvaluation:')) { /* widen cases or realign eval */ }
  else throw e;
}

When it happens

Trigger: Passing a filtered/excluded testCasesWithFiles list that omits a case the evaluation actually ran; cases renamed or deleted on disk after the evaluation was produced; comparing evaluations across git revisions where the case set changed.

Common situations: Running comparison with --filter/--exclude that drops an evaluated case; renaming a case file between eval and compare; stale evaluation JSON referencing old case identities.

Related errors

• Unknown role: ${msg.role as string}
• Delegated child checkpoint metadata is missing or invalid
• Eval "${this.evalName}": cannot use both .check() and .judge
• Eval "${this.evalName}" requires either .check() or .judge()
• Eval "${this.evalName}" uses .judge() but no .model() was se

AI-assisted analysis of n8n-io/n8n@5ac6606e81 (2026-08-12). Data as JSON: /api/errors/f1d049b70d7ab83b. Report an issue: GitHub.