n8n-io/n8n · error · Error

LANGSMITH_API_KEY is not set — the AIA arm needs LangSmith a

Error message

LANGSMITH_API_KEY is not set — the AIA arm needs LangSmith access to sum builder cost.

What it means

Thrown by langsmithConfig() in build-cost-report.ts:106 when configFor() returns no apiKey. The Instance AI Assistant (AIA) arm sums builder cost by joining each test case's build threads to root runs in the LangSmith trace project; without an API key, that thread join cannot run. The key is read via configFor() which sources LANGSMITH_ENDPOINT / LANGSMITH_API_KEY. Note this is resolved lazily — only arms that need the thread join (those with threadIds but no persisted buildCostUsdPerRun) trigger it; a persisted-spend-only comparison runs without credentials.

Source

Thrown at packages/@n8n/instance-ai/evaluations/cli/build-cost-report.ts:106

/** LangSmith request with 429/5xx backoff — a full-run join fires ~100 queries,
 *  which trips the API's rate limit without pacing. Honors Retry-After. */
async function lsFetch(url: string, init: RequestInit): Promise<Response> {
	let delayMs = 2_000;
	for (let attempt = 1; ; attempt++) {
		const res = await fetch(url, init);
		if (res.ok || attempt >= 6 || (res.status !== 429 && res.status < 500)) return res;
		const retryAfter = Number(res.headers.get('retry-after'));
		const waitMs = Number.isFinite(retryAfter) && retryAfter > 0 ? retryAfter * 1_000 : delayMs;
		await new Promise((resolve) => setTimeout(resolve, waitMs));
		delayMs = Math.min(delayMs * 2, 30_000);
	}
}

async function langsmithConfig(): Promise<LangSmithConfig> {
	const { apiUrl, apiKey } = configFor();
	if (!apiKey) {
		throw new Error(
			'LANGSMITH_API_KEY is not set — the AIA arm needs LangSmith access to sum builder cost.',
		);
	}
	const workspaceId = await resolveEvalWorkspaceId();
	return {
		apiUrl,
		headers: {
			'x-api-key': apiKey,
			'Content-Type': 'application/json',
			...(workspaceId ? { 'X-Tenant-Id': workspaceId } : {}),
		},
	};
}

async function resolveTraceProjectId(ls: LangSmithConfig, projectName: string): Promise<string> {
	const res = await lsFetch(
		`${ls.apiUrl}/api/v1/sessions?name=${encodeURIComponent(projectName)}`,
		{

View on GitHub (pinned to 5ac6606e81)

Solutions

Export LANGSMITH_API_KEY (and LANGSMITH_ENDPOINT if non-default) before running the report: `export LANGSMITH_API_KEY=lsv2_pt_...`.
Load from dotenv: `dotenvx run -f .env.local -- pnpm tsx evaluations/cli/build-cost-report.ts --results ...`.
If you only want persisted-spend arms (MCP runs with buildCostUsdPerRun), ensure all input files have that field and no threadIds — then the key is never needed.
For --probe-thread mode, the key is always required; set it unconditionally.

Example fix

// before
pnpm tsx evaluations/cli/build-cost-report.ts --results aia-run/eval-results.json
// after
export LANGSMITH_API_KEY=lsv2_pt_...
pnpm tsx evaluations/cli/build-cost-report.ts --results aia-run/eval-results.json

Defensive patterns

Strategy: validation

Validate before calling

// Before running build-cost-report for an AIA (thread-join) arm:
if (!process.env.LANGSMITH_API_KEY) {
  throw new Error('LANGSMITH_API_KEY required for thread-join (AIA) cost arms.');
}

Prevention

Always source .env.local when running build-cost-report: dotenvx run -f .env.local -- ....
For AIA arms, set LANGSMITH_API_KEY unconditionally even if you think only MCP arms are present.
Use --probe-thread as a connectivity smoke test before a full report run.

When it happens

Trigger: Running build-cost-report.ts with at least one --results file whose test cases carry threadIds (AIA-origin) but no buildCostUsdPerRun field, and LANGSMITH_API_KEY is not set in the environment. Also triggered by --probe-thread <id> (which always needs LangSmith).

Common situations: Comparing an AIA eval run against an MCP run without sourcing .env.local, or in CI where the LangSmith secret is missing or scoped to a different job. The lazy resolution means the error surfaces only when a thread-join arm is encountered, which can be confusing.

Related errors

AI-assisted analysis of n8n-io/n8n@5ac6606e81 (2026-08-12). Data as JSON: /api/errors/8ceff11cfc0031f9. Report an issue: GitHub.