{"record":{"id":"2f1ce3fc72d43281","repo":"n8n-io/n8n","slug":"langsmith-mode-requires-dataset-and-does-not-s","errorCode":null,"errorMessage":"LangSmith mode requires `--dataset` and does not support `--prompt`, `--prompts-csv`, or `--test-case`","messagePattern":"LangSmith mode requires `--dataset` and does not support `--prompt`, `--prompts-csv`, or `--test-case`","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/@n8n/ai-workflow-builder.ee/evaluations/cli/index.ts","lineNumber":472,"sourceCode":"\t\t\t\t},\n\t\t\t},\n\t\t];\n\t\treturn args.maxExamples ? testCases.slice(0, args.maxExamples) : testCases;\n\t}\n\n\t// Default: use bundled test cases\n\tconst defaultCases = loadDefaultTestCases();\n\treturn args.maxExamples ? defaultCases.slice(0, args.maxExamples) : defaultCases;\n}\n\n/**\n * Main entry point for v2 evaluation CLI.\n */\nexport async function runV2Evaluation(): Promise<void> {\n\tconst args = parseEvaluationArgs();\n\n\tif (args.backend === 'langsmith' && (args.prompt || args.promptsCsv || args.testCase)) {\n\t\tthrow new Error(\n\t\t\t'LangSmith mode requires `--dataset` and does not support `--prompt`, `--prompts-csv`, or `--test-case`',\n\t\t);\n\t}\n\n\t// Setup environment with per-stage model configuration\n\tconst logger = createLogger(args.verbose);\n\tconst stageModels = argsToStageModels(args);\n\n\tconst env = await setupTestEnvironment(stageModels, logger);\n\n\t// Validate LangSmith client early if langsmith backend is requested\n\tif (args.backend === 'langsmith' && !env.lsClient) {\n\t\tthrow new Error('LangSmith client not initialized - check LANGSMITH_API_KEY');\n\t}\n\n\t// Create workflow generator based on agent type\n\tconst generateWorkflow =\n\t\targs.agent === AGENT_TYPES.CODE_BUILDER","sourceCodeStart":454,"sourceCodeEnd":490,"githubUrl":"https://github.com/n8n-io/n8n/blob/5ac6606e81f67bb9534255570cd4e86fd8101eee/packages/@n8n/ai-workflow-builder.ee/evaluations/cli/index.ts#L454-L490","documentation":"Thrown by the v2 evaluation CLI entry point when `--backend langsmith` is combined with `--prompt`, `--prompts-csv`, or `--test-case`. LangSmith mode pulls its inputs from a named LangSmith dataset (via `--dataset`), so the CLI flags that supply inline prompts are mutually exclusive with it. The check exists so the run fails fast with a clear message instead of silently ignoring the user's prompt.","triggerScenarios":"Invoking `runV2Evaluation()` after `parseEvaluationArgs()` returned `backend === 'langsmith'` AND any of `prompt`, `promptsCsv`, or `testCase` is set. Concretely: `pnpm eval --backend langsmith --prompt \"build a slack notifier\"`, `--backend langsmith --prompts-csv cases.csv`, or `--backend langsmith --test-case foo`.","commonSituations":"Developer copies a local-mode command and just appends `--backend langsmith`, forgetting to switch the input source. Or CI/automation sets `LANGSMITH_*` env and reuses the same args array used for local runs. Mixing LangSmith dataset semantics with the inline-prompt workflow is the typical slip.","solutions":["Drop `--prompt`, `--prompts-csv`, and `--test-case` and supply `--dataset <dataset-name>` instead so inputs come from LangSmith.","If you want to run inline prompts, keep them and remove `--backend langsmith` (run with the default/local backend).","Double-check the arg parser / shell script: a stray `--prompt` from an earlier invocation left in `$@` triggers this even when you think you passed `--dataset`."],"exampleFix":"// before\npnpm eval --backend langsmith --prompt \"build a slack notifier\"\n// after\npnpm eval --backend langsmith --dataset my-dataset-name","handlingStrategy":"validation","validationCode":"import { parseEvaluationArgs } from './cli';\n\nfunction assertLangsmithArgsOk(argv: string[]): void {\n  // mirror parseEvaluationArgs or accept its result\n  const has = (f: string) => argv.includes(f);\n  const backendLangsmith =\n    argv.some((a, i) => a === '--backend' && argv[i + 1] === 'langsmith') ||\n    argv.some((a) => a.startsWith('--backend=langsmith'));\n  const incompatible = has('--prompt') || has('--prompts-csv') || has('--test-case');\n  if (backendLangsmith && incompatible) {\n    throw new Error(\n      'LangSmith mode requires --dataset; remove --prompt/--prompts-csv/--test-case or drop --backend langsmith',\n    );\n  }\n  if (backendLangsmith && !has('--dataset')) {\n    throw new Error('LangSmith mode requires --dataset <name>');\n  }\n}\n// run before invoking the CLI\nassertLangsmithArgsOk(process.argv.slice(2));","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Wrap your eval invocation in a shell script that branches on `$EVAL_BACKEND` and only ever passes `--prompt`/`--prompts-csv`/`--test-case` for the local backend.","Treat `--backend langsmith` and the inline-prompt flags as mutually exclusive in your team's CLI docs and CI arg validation.","Unit-test the arg validator so a regression in the mutually-exclusive rule is caught before merge."],"tags":["langsmith","cli","evaluations","validation","configuration"],"backgroundTag":null,"analyzedSha":"5ac6606e81f67bb9534255570cd4e86fd8101eee","analyzedAt":"2026-08-12T05:26:35.080Z","schemaVersion":2},"datasetVersion":"2026-08-12T18:17:37.767Z"}