n8n-io/n8n · error · Error

Failed to parse judge response as JSON: ${jsonStr.slice(0, 1

Error message

Failed to parse judge response as JSON: ${jsonStr.slice(0, 100)}...

What it means

The responder evaluator asks the judge LLM for a JSON object matching `ResponderJudgeResult` (relevance/accuracy/completeness/clarity/tone/criteriaMatch/forbiddenPhrases dimensions + overallScore/summary). `parseJudgeResponse` first strips a markdown ```json fence if present, then `JSON.parse`s. If parsing fails, the first 100 chars of the judge's raw output are included in the thrown message to aid diagnosis.

Source

Thrown at packages/@n8n/ai-workflow-builder.ee/evaluations/evaluators/responder/index.ts:63

}

function isResponderContext(ctx: EvaluationContext): ctx is ResponderEvaluationContext {
	return (
		'responderOutput' in ctx &&
		typeof (ctx as ResponderEvaluationContext).responderOutput === 'string' &&
		'responderEvals' in ctx &&
		typeof (ctx as ResponderEvaluationContext).responderEvals === 'object'
	);
}

function parseJudgeResponse(content: string): ResponderJudgeResult {
	// Extract JSON from markdown code block if present
	const jsonMatch = content.match(/```(?:json)?\s*([\s\S]*?)```/);
	const jsonStr = jsonMatch ? jsonMatch[1].trim() : content.trim();
	try {
		return JSON.parse(jsonStr) as ResponderJudgeResult;
	} catch {
		throw new Error(`Failed to parse judge response as JSON: ${jsonStr.slice(0, 100)}...`);
	}
}

const DIMENSION_KEYS = [
	'relevance',
	'accuracy',
	'completeness',
	'clarity',
	'tone',
	'criteriaMatch',
	'forbiddenPhrases',
] as const;

const fb = (metric: string, score: number, kind: Feedback['kind'], comment?: string): Feedback => ({
	evaluator: EVALUATOR_NAME,
	metric,
	score,
	kind,

View on GitHub (pinned to 5ac6606e81)

Solutions

1. Use `withStructuredOutput` / tool-calling for the judge (as the base evaluator chain does) so the model is forced to emit schema-conformant output instead of free-form JSON.
2. Tighten the judge system prompt to forbid any text outside the JSON/code fence, and switch to a model that follows those instructions reliably.
3. If you cannot change the evaluator, retry with `numJudges > 1` so a single malformed response is less likely to fail the whole run, and log the raw `content` for inspection.

Example fix

// before
function parseJudgeResponse(content: string): ResponderJudgeResult {
  const jsonStr = ... ;
  try { return JSON.parse(jsonStr) as ResponderJudgeResult; }
  catch { throw new Error(`Failed to parse judge response as JSON: ${jsonStr.slice(0, 100)}...`); }
}
// after - tolerate prose-then-JSON and validate shape
function parseJudgeResponse(content: string): ResponderJudgeResult {
  const match = content.match(/\{[\s\S]*\}/);
  const parsed = JSON.parse(match ? match[0] : content) as ResponderJudgeResult;
  for (const k of DIMENSION_KEYS) if (!parsed[k]) throw new Error(`judge JSON missing dimension ${k}`);
  return parsed;
}

function looksLikeCompleteJson(s: string): boolean {
  const t = s.trim();
  if (!t.startsWith('{') || !t.endsWith('}')) return false;
  try { JSON.parse(t); return true; } catch { return false; }
}
// only call parseJudgeResponse after this returns true; otherwise surface raw content for debugging

function isResponderJudgeResult(v: unknown): v is ResponderJudgeResult {
  if (!v || typeof v !== 'object') return false;
  const r = v as Record<string, unknown>;
  return ['relevance','accuracy','completeness','clarity','tone','criteriaMatch','forbiddenPhrases']
    .every((k) => typeof r[k]?.score === 'number');
}

try {
  return parseJudgeResponse(content);
} catch (e) {
  if (e instanceof Error && e.message.startsWith('Failed to parse judge response as JSON')) {
    logger.warn(`judge produced non-JSON output (first 100 chars): ${content.slice(0, 100)}`);
    // fall back to a neutral score so one bad judge does not sink the example
    return NEUTRAL_RESPONDER_RESULT;
  }
  throw e;
}

When it happens

Trigger: Judge model returns prose instead of JSON, returns JSON with trailing commas / comments / single quotes, wraps the object in extra text outside the code fence, or returns an empty string. The fence-stripping regex only matches a single ```json ... ``` block.

Common situations: Switching to a smaller/weaker judge model that doesn't follow output-format instructions; a prompt change that dropped the 'respond only with JSON' instruction; the model prepending an explanation like 'Here is the evaluation:' before the fence; rate-limit/truncation cutting the JSON mid-stream.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• LLM doesn't support binding tools
• Failed to parse ${label} at ${path}: ${msg}
• Failed to parse query: ${(error as Error).message}
• The value in the "Extra Body" field is not valid JSON
• The value in the "Extra Body" field must be a JSON object

AI-assisted analysis of n8n-io/n8n@5ac6606e81 (2026-08-12). Data as JSON: /api/errors/98900ef9bc403df0. Report an issue: GitHub.