{"record":{"id":"89f24f31d419ab87","repo":"n8n-io/n8n","slug":"invalid-discovery-test-case-filepath-issues","errorCode":null,"errorMessage":"Invalid discovery test case ${filePath}: ${issues}","messagePattern":"Invalid discovery test case (.+?): (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/@n8n/instance-ai/evaluations/data/discovery/index.ts","lineNumber":79,"sourceCode":"\tfileSlug: string;\n}\n\nfunction parseTestCaseFile(filePath: string): DiscoveryTestCase {\n\tconst content = readFileSync(filePath, 'utf-8');\n\tlet raw: unknown;\n\ttry {\n\t\traw = JSON.parse(content);\n\t} catch (error) {\n\t\tthrow new Error(\n\t\t\t`Failed to parse discovery test case ${filePath}: ${error instanceof Error ? error.message : String(error)}`,\n\t\t);\n\t}\n\tconst parsed = discoveryTestCaseSchema.safeParse(raw);\n\tif (!parsed.success) {\n\t\tconst issues = parsed.error.issues\n\t\t\t.map((issue) => `${issue.path.join('.') || '(root)'}: ${issue.message}`)\n\t\t\t.join('; ');\n\t\tthrow new Error(`Invalid discovery test case ${filePath}: ${issues}`);\n\t}\n\treturn parsed.data;\n}\n\nfunction parseSubstringList(value: string | undefined): string[] {\n\tif (!value) return [];\n\treturn value\n\t\t.split(',')\n\t\t.map((s) => s.trim().toLowerCase())\n\t\t.filter((s) => s.length > 0);\n}\n\nfunction getJsonFiles(filter?: string, exclude?: string): string[] {\n\tconst dir = __dirname;\n\tlet files = readdirSync(dir).filter((f) => f.endsWith('.json'));\n\n\tconst includeTokens = parseSubstringList(filter);\n\tif (includeTokens.length > 0) {","sourceCodeStart":61,"sourceCodeEnd":97,"githubUrl":"https://github.com/n8n-io/n8n/blob/5ac6606e81f67bb9534255570cd4e86fd8101eee/packages/@n8n/instance-ai/evaluations/data/discovery/index.ts#L61-L97","documentation":"Thrown by `parseTestCaseFile()` when `JSON.parse` succeeded but the zod schema `discoveryTestCaseSchema.safeParse` rejected the object. The schema is strict (`.strict()` — unknown keys fail) and requires: `id` (non-empty string), `userMessage` (non-empty string), optional `instanceState` with a discriminated `localGateway` union and `browserAvailable` boolean, and `expectedToolInvocations` which must itself be non-empty in at least one of `anyOf/noneOf/anyOfToolCalls/allOfToolCalls/noneOfToolCalls`. The message joins all zod issues into a `path: message; ...` string so each failing field is named.","triggerScenarios":"Missing required `id` or `userMessage`. A typo'd key like `expectedToolInvocation` (missing trailing `s`) which `.strict()` rejects as unknown. An empty `anyOf: []` (the schema uses `.min(1)` on every list). An `instanceState.localGateway.status` value not in the discriminated union (`connected`/`disabledGlobally`/`disconnected`/`disabled`). A `maxSteps` set to 0 or a non-integer (schema requires `.int().positive()`). A `localGateway.capabilities` value that isn't a string array.","commonSituations":"Authoring a discovery case and forgetting one of the required fields. Adding a new expectation key the schema doesn't know about yet. Copying a workflow-case file shape into a discovery case — they have different schemas.","solutions":["Read the issue list in the message — each entry is `path: message`; fix each cited field.","Open data/discovery/index.ts:28-56 to see the authoritative schema (it is the source of truth, not a separate doc).","Common fixes: ensure `id` and `userMessage` are present and non-empty; ensure every list under `expectedToolInvocations` has at least one entry; remove unknown keys (strict mode rejects them).","Re-run; the loader re-reads the file each time, no build step needed for case JSON."],"exampleFix":"// before (case.json)\n{\n  \"id\": \"slack-setup\",\n  \"userMessage\": \"set up slack\",\n  \"expectedToolInvocations\": { \"anyOf\": [] }\n}\n// after\n{\n  \"id\": \"slack-setup\",\n  \"userMessage\": \"set up slack\",\n  \"expectedToolInvocations\": { \"anyOf\": [\"create_credential\"] }\n}","handlingStrategy":"validation","validationCode":"import { discoveryTestCaseSchema } from './index';\nfunction validateCaseFile(filePath: string): DiscoveryTestCase {\n  const raw = JSON.parse(readFileSync(filePath, 'utf-8'));\n  const parsed = discoveryTestCaseSchema.safeParse(raw);\n  if (!parsed.success) {\n    const issues = parsed.error.issues.map((i) => `${i.path.join('.') || '(root)'}: ${i.message}`).join('; ');\n    throw new Error(`${filePath}: ${issues}`);\n  }\n  return parsed.data;\n}","typeGuard":"function isValidDiscoveryTestCase(raw: unknown): raw is DiscoveryTestCase {\n  return discoveryTestCaseSchema.safeParse(raw).success;\n}","tryCatchPattern":null,"preventionTips":["Use the exported `discoveryTestCaseSchema` as the single source of truth; don't define a parallel type by hand.","Add a pre-commit hook that runs `discoveryTestCaseSchema.parse` over changed case files.","When extending the schema, add a sample case file in the same PR so the schema is exercised."],"tags":["discovery","test-case-config","zod","schema","validation","authoring"],"backgroundTag":null,"analyzedSha":"5ac6606e81f67bb9534255570cd4e86fd8101eee","analyzedAt":"2026-08-12T05:26:35.080Z","schemaVersion":2},"datasetVersion":"2026-08-12T13:17:24.610Z"}