n8n-io/n8n · error · Error

LangSmith mode requires `--dataset` and does not support `--

Error message

LangSmith mode requires `--dataset` and does not support `--prompt`, `--prompts-csv`, or `--test-case`

What it means

Thrown by the v2 evaluation CLI entry point when `--backend langsmith` is combined with `--prompt`, `--prompts-csv`, or `--test-case`. LangSmith mode pulls its inputs from a named LangSmith dataset (via `--dataset`), so the CLI flags that supply inline prompts are mutually exclusive with it. The check exists so the run fails fast with a clear message instead of silently ignoring the user's prompt.

Source

Thrown at packages/@n8n/ai-workflow-builder.ee/evaluations/cli/index.ts:472

				},
			},
		];
		return args.maxExamples ? testCases.slice(0, args.maxExamples) : testCases;
	}

	// Default: use bundled test cases
	const defaultCases = loadDefaultTestCases();
	return args.maxExamples ? defaultCases.slice(0, args.maxExamples) : defaultCases;
}

/**
 * Main entry point for v2 evaluation CLI.
 */
export async function runV2Evaluation(): Promise<void> {
	const args = parseEvaluationArgs();

	if (args.backend === 'langsmith' && (args.prompt || args.promptsCsv || args.testCase)) {
		throw new Error(
			'LangSmith mode requires `--dataset` and does not support `--prompt`, `--prompts-csv`, or `--test-case`',
		);
	}

	// Setup environment with per-stage model configuration
	const logger = createLogger(args.verbose);
	const stageModels = argsToStageModels(args);

	const env = await setupTestEnvironment(stageModels, logger);

	// Validate LangSmith client early if langsmith backend is requested
	if (args.backend === 'langsmith' && !env.lsClient) {
		throw new Error('LangSmith client not initialized - check LANGSMITH_API_KEY');
	}

	// Create workflow generator based on agent type
	const generateWorkflow =
		args.agent === AGENT_TYPES.CODE_BUILDER

View on GitHub (pinned to 5ac6606e81)

Solutions

1. Drop `--prompt`, `--prompts-csv`, and `--test-case` and supply `--dataset <dataset-name>` instead so inputs come from LangSmith.
2. If you want to run inline prompts, keep them and remove `--backend langsmith` (run with the default/local backend).
3. Double-check the arg parser / shell script: a stray `--prompt` from an earlier invocation left in `$@` triggers this even when you think you passed `--dataset`.

Example fix

// before
pnpm eval --backend langsmith --prompt "build a slack notifier"
// after
pnpm eval --backend langsmith --dataset my-dataset-name

import { parseEvaluationArgs } from './cli';

function assertLangsmithArgsOk(argv: string[]): void {
  // mirror parseEvaluationArgs or accept its result
  const has = (f: string) => argv.includes(f);
  const backendLangsmith =
    argv.some((a, i) => a === '--backend' && argv[i + 1] === 'langsmith') ||
    argv.some((a) => a.startsWith('--backend=langsmith'));
  const incompatible = has('--prompt') || has('--prompts-csv') || has('--test-case');
  if (backendLangsmith && incompatible) {
    throw new Error(
      'LangSmith mode requires --dataset; remove --prompt/--prompts-csv/--test-case or drop --backend langsmith',
    );
  }
  if (backendLangsmith && !has('--dataset')) {
    throw new Error('LangSmith mode requires --dataset <name>');
  }
}
// run before invoking the CLI
assertLangsmithArgsOk(process.argv.slice(2));

When it happens

Trigger: Invoking `runV2Evaluation()` after `parseEvaluationArgs()` returned `backend === 'langsmith'` AND any of `prompt`, `promptsCsv`, or `testCase` is set. Concretely: `pnpm eval --backend langsmith --prompt "build a slack notifier"`, `--backend langsmith --prompts-csv cases.csv`, or `--backend langsmith --test-case foo`.

Common situations: Developer copies a local-mode command and just appends `--backend langsmith`, forgetting to switch the input source. Or CI/automation sets `LANGSMITH_*` env and reuses the same args array used for local runs. Mixing LangSmith dataset semantics with the inline-prompt workflow is the typical slip.

Related errors

• LangSmith client not initialized - check LANGSMITH_API_KEY
• No valid checks after filtering. Requested: ${options.checks
• No examples found in dataset "${datasetName}"
• Model ID is required
• Invalid delegate sub-agent tool name "${name}": must start w

AI-assisted analysis of n8n-io/n8n@5ac6606e81 (2026-08-12). Data as JSON: /api/errors/2f1ce3fc72d43281. Report an issue: GitHub.