{"record":{"id":"f1d049b70d7ab83b","repo":"n8n-io/n8n","slug":"bucketfromevaluation-no-fileslug-for-test-case","errorCode":null,"errorMessage":"bucketFromEvaluation: no fileSlug for test case \"${caseDisplayPrompt(tc.testCase, tc.runs[0]?.transcript).slice(0, 60)}\"","messagePattern":"bucketFromEvaluation: no fileSlug for test case \"(.+?)\"","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/@n8n/instance-ai/evaluations/comparison/bucket-from-evaluation.ts","lineNumber":40,"sourceCode":" * Looks up `fileSlug` by test case reference rather than array index — the\n * comparison key depends on getting the right slug, and zipping by index\n * silently miscompares if anything ever reorders the aggregate.\n */\nexport function bucketFromEvaluation(\n\tevaluation: MultiRunEvaluation,\n\ttestCasesWithFiles: WorkflowTestCaseWithFile[],\n\texperimentName: string,\n): ExperimentBucket {\n\tconst slugByTestCase = new Map(\n\t\ttestCasesWithFiles.map(({ testCase, fileSlug }) => [testCase, fileSlug]),\n\t);\n\tconst evaluationUnits = new Map<string, EvaluationUnitCounts>();\n\tconst failureCategoryTotals: Record<string, number> = {};\n\tlet trialTotal = 0;\n\tfor (const tc of evaluation.testCases) {\n\t\tconst fileSlug = slugByTestCase.get(tc.testCase);\n\t\tif (!fileSlug) {\n\t\t\tthrow new Error(\n\t\t\t\t`bucketFromEvaluation: no fileSlug for test case \"${caseDisplayPrompt(tc.testCase, tc.runs[0]?.transcript).slice(0, 60)}\"`,\n\t\t\t);\n\t\t}\n\t\tfor (const sa of tc.executionScenarios) {\n\t\t\tconst failureCategories: Record<string, number> = {};\n\t\t\tfor (const sr of sa.runs) {\n\t\t\t\t// Verifier-incomplete runs carry no verdict — not a trial.\n\t\t\t\tif (sr.incomplete) continue;\n\t\t\t\ttrialTotal++;\n\t\t\t\tif (!sr.success && sr.failureCategory) {\n\t\t\t\t\tfailureCategories[sr.failureCategory] = (failureCategories[sr.failureCategory] ?? 0) + 1;\n\t\t\t\t\tfailureCategoryTotals[sr.failureCategory] =\n\t\t\t\t\t\t(failureCategoryTotals[sr.failureCategory] ?? 0) + 1;\n\t\t\t\t}\n\t\t\t}\n\t\t\tevaluationUnits.set(scenarioUnitKey(fileSlug, sa.scenario.name), {\n\t\t\t\tkind: 'scenario',\n\t\t\t\ttestCaseFile: fileSlug,","sourceCodeStart":22,"sourceCodeEnd":58,"githubUrl":"https://github.com/n8n-io/n8n/blob/5ac6606e81f67bb9534255570cd4e86fd8101eee/packages/@n8n/instance-ai/evaluations/comparison/bucket-from-evaluation.ts#L22-L58","documentation":"bucketFromEvaluation joins an evaluation result to the on-disk case files via a testCase→fileSlug map built from testCasesWithFiles. If an evaluation.testCases entry has no matching testCase object in that map, the comparison cannot attribute outcomes to a slug, so it throws. The message surfaces a truncated prompt so you can identify the orphan case.","triggerScenarios":"Passing a filtered/excluded testCasesWithFiles list that omits a case the evaluation actually ran; cases renamed or deleted on disk after the evaluation was produced; comparing evaluations across git revisions where the case set changed.","commonSituations":"Running comparison with --filter/--exclude that drops an evaluated case; renaming a case file between eval and compare; stale evaluation JSON referencing old case identities.","solutions":["Pass the full set of loaded cases (no filter) to bucketFromEvaluation so every evaluated case has a slug.","Re-run the evaluation if cases were renamed, so testCase identities match the current files.","Filter the evaluation's testCases down before calling bucketFromEvaluation so orphans are dropped intentionally."],"exampleFix":"// before: const bucket = bucketFromEvaluation(eval, filteredCases, name);\n// after:  const aligned = eval.testCases.filter(tc => loadedSet.has(tc.testCase));\n//        const bucket = bucketFromEvaluation({ ...eval, testCases: aligned }, loadedCases, name);","handlingStrategy":"validation","validationCode":"const slugByCase = new Map(testCasesWithFiles.map(({ testCase, fileSlug }) => [testCase, fileSlug]));\nconst orphans = evaluation.testCases.filter(tc => !slugByCase.has(tc.testCase));\nif (orphans.length) throw new Error(`evaluation has ${orphans.length} case(s) absent from testCasesWithFiles`);","typeGuard":"const isWorkflowTestCaseWithFile = (v: unknown): v is { testCase: unknown; fileSlug: string } =>\n  typeof v === 'object' && v !== null && typeof (v as any).fileSlug === 'string' && 'testCase' in (v as object);","tryCatchPattern":"try { return bucketFromEvaluation(eval, cases, name); }\ncatch (e) {\n  if (e instanceof Error && e.message.startsWith('bucketFromEvaluation:')) { /* widen cases or realign eval */ }\n  else throw e;\n}","preventionTips":["Pass the unfiltered case set to bucketFromEvaluation.","Re-run evaluations after renaming case files.","Filter the evaluation's testCases to align with the cases you pass."],"tags":["eval","comparison","data-integrity"],"backgroundTag":null,"analyzedSha":"5ac6606e81f67bb9534255570cd4e86fd8101eee","analyzedAt":"2026-08-12T05:26:35.080Z","schemaVersion":2},"datasetVersion":"2026-08-12T18:17:37.767Z"}