n8n-io/n8n · error · Error

Invalid discovery test case ${filePath}: ${issues}

Error message

Invalid discovery test case ${filePath}: ${issues}

What it means

Thrown by `parseTestCaseFile()` when `JSON.parse` succeeded but the zod schema `discoveryTestCaseSchema.safeParse` rejected the object. The schema is strict (`.strict()` — unknown keys fail) and requires: `id` (non-empty string), `userMessage` (non-empty string), optional `instanceState` with a discriminated `localGateway` union and `browserAvailable` boolean, and `expectedToolInvocations` which must itself be non-empty in at least one of `anyOf/noneOf/anyOfToolCalls/allOfToolCalls/noneOfToolCalls`. The message joins all zod issues into a `path: message; ...` string so each failing field is named.

Source

Thrown at packages/@n8n/instance-ai/evaluations/data/discovery/index.ts:79

	fileSlug: string;
}

function parseTestCaseFile(filePath: string): DiscoveryTestCase {
	const content = readFileSync(filePath, 'utf-8');
	let raw: unknown;
	try {
		raw = JSON.parse(content);
	} catch (error) {
		throw new Error(
			`Failed to parse discovery test case ${filePath}: ${error instanceof Error ? error.message : String(error)}`,
		);
	}
	const parsed = discoveryTestCaseSchema.safeParse(raw);
	if (!parsed.success) {
		const issues = parsed.error.issues
			.map((issue) => `${issue.path.join('.') || '(root)'}: ${issue.message}`)
			.join('; ');
		throw new Error(`Invalid discovery test case ${filePath}: ${issues}`);
	}
	return parsed.data;
}

function parseSubstringList(value: string | undefined): string[] {
	if (!value) return [];
	return value
		.split(',')
		.map((s) => s.trim().toLowerCase())
		.filter((s) => s.length > 0);
}

function getJsonFiles(filter?: string, exclude?: string): string[] {
	const dir = __dirname;
	let files = readdirSync(dir).filter((f) => f.endsWith('.json'));

	const includeTokens = parseSubstringList(filter);
	if (includeTokens.length > 0) {

View on GitHub (pinned to 5ac6606e81)

Solutions

1. Read the issue list in the message — each entry is `path: message`; fix each cited field.
2. Open data/discovery/index.ts:28-56 to see the authoritative schema (it is the source of truth, not a separate doc).
3. Common fixes: ensure `id` and `userMessage` are present and non-empty; ensure every list under `expectedToolInvocations` has at least one entry; remove unknown keys (strict mode rejects them).
4. Re-run; the loader re-reads the file each time, no build step needed for case JSON.

Example fix

// before (case.json)
{
  "id": "slack-setup",
  "userMessage": "set up slack",
  "expectedToolInvocations": { "anyOf": [] }
}
// after
{
  "id": "slack-setup",
  "userMessage": "set up slack",
  "expectedToolInvocations": { "anyOf": ["create_credential"] }
}

import { discoveryTestCaseSchema } from './index';
function validateCaseFile(filePath: string): DiscoveryTestCase {
  const raw = JSON.parse(readFileSync(filePath, 'utf-8'));
  const parsed = discoveryTestCaseSchema.safeParse(raw);
  if (!parsed.success) {
    const issues = parsed.error.issues.map((i) => `${i.path.join('.') || '(root)'}: ${i.message}`).join('; ');
    throw new Error(`${filePath}: ${issues}`);
  }
  return parsed.data;
}

function isValidDiscoveryTestCase(raw: unknown): raw is DiscoveryTestCase {
  return discoveryTestCaseSchema.safeParse(raw).success;
}

When it happens

Trigger: Missing required `id` or `userMessage`. A typo'd key like `expectedToolInvocation` (missing trailing `s`) which `.strict()` rejects as unknown. An empty `anyOf: []` (the schema uses `.min(1)` on every list). An `instanceState.localGateway.status` value not in the discriminated union (`connected`/`disabledGlobally`/`disconnected`/`disabled`). A `maxSteps` set to 0 or a non-integer (schema requires `.int().positive()`). A `localGateway.capabilities` value that isn't a string array.

Common situations: Authoring a discovery case and forgetting one of the required fields. Adding a new expectation key the schema doesn't know about yet. Copying a workflow-case file shape into a discovery case — they have different schemas.

Related errors

• Failed to parse discovery test case ${filePath}: ${error ins
• Tool "${this.name}" requires an input schema
• No prompt found in inputs - expected "prompt" string or "mes
• VALIDATION_ERROR
• VALIDATION_ERROR

AI-assisted analysis of n8n-io/n8n@5ac6606e81 (2026-08-12). Data as JSON: /api/errors/89f24f31d419ab87. Report an issue: GitHub.