n8n-io/n8n · error · Error

--set-kind must be "regression" or "capability_gap"

Error message

--set-kind must be "regression" or "capability_gap"

What it means

The --set-kind flag for langtracer-push controls the test-set classification assigned to created cases in the lang-tracer suite. It accepts exactly two enum values: 'regression' (default) or 'capability_gap'. Any other value is rejected because the lang-tracer API expects these exact strings — an invalid value would fail server-side or silently miscategorize cases.

Source

Thrown at packages/@n8n/instance-ai/evaluations/cli/langtracer-push.ts:92

				result.filter = nextArg(argv, i, arg);
				i += 2;
				break;
			case '--exclude':
				result.exclude = nextArg(argv, i, arg);
				i += 2;
				break;
			case '--tier':
				result.tier = nextArg(argv, i, arg);
				i += 2;
				break;
			case '--changed':
				result.changed = true;
				i += 1;
				break;
			case '--set-kind': {
				const value = nextArg(argv, i, arg);
				if (value !== 'regression' && value !== 'capability_gap') {
					throw new Error('--set-kind must be "regression" or "capability_gap"');
				}
				result.setKind = value;
				i += 2;
				break;
			}
			case '--contains-user-data':
				result.synthetic = false;
				i += 1;
				break;
			case '--dry-run':
				result.dryRun = true;
				i += 1;
				break;
			case '-h':
			case '--help':
				return { helpRequested: true };
			default:
				if (arg.startsWith('--')) {

View on GitHub (pinned to 5ac6606e81)

Solutions

1. Use exactly 'regression' or 'capability_gap' (lowercase, underscore-separated)
2. Omit the flag to use the default value 'regression'

Example fix

# before
pnpm eval:langtracer-push --suite baseline --changed --set-kind regression_test

# after
pnpm eval:langtracer-push --suite baseline --changed --set-kind regression

const VALID_SET_KINDS = ['regression', 'capability_gap'] as const;
function parseSetKind(raw: string): typeof VALID_SET_KINDS[number] {
  if (!VALID_SET_KINDS.includes(raw as never)) {
    throw new Error(`--set-kind must be one of: ${VALID_SET_KINDS.join(', ')}`);
  }
  return raw as typeof VALID_SET_KINDS[number];
}

function isSetKind(v: unknown): v is 'regression' | 'capability_gap' {
  return v === 'regression' || v === 'capability_gap';
}

When it happens

Trigger: Running `pnpm eval:langtracer-push --suite baseline --changed --set-kind regression_test` or any value other than 'regression' or 'capability_gap'. The check at line 91 fires immediately after nextArg returns the value.

Common situations: Typo or variation (e.g. 'regression_test', 'Regression', 'capability-gap' with a hyphen). Guessing the enum value. Using a camelCase variant like 'capabilityGap'.

Related errors

• Unknown flag: ${arg.split('=', 1)[0]} (use --help)
• --suite <slug|id> is required
• select cases to push: pass <slugs...>, --changed, --filter,
• Missing value for ${flag}
• --max-attempts must be >= 1

AI-assisted analysis of n8n-io/n8n@5ac6606e81 (2026-08-12). Data as JSON: /api/errors/346cb573e4f94943. Report an issue: GitHub.