{"record":{"id":"98900ef9bc403df0","repo":"n8n-io/n8n","slug":"failed-to-parse-judge-response-as-json-jsonstr","errorCode":null,"errorMessage":"Failed to parse judge response as JSON: ${jsonStr.slice(0, 100)}...","messagePattern":"Failed to parse judge response as JSON: (.+?)\\.\\.\\.","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/@n8n/ai-workflow-builder.ee/evaluations/evaluators/responder/index.ts","lineNumber":63,"sourceCode":"}\n\nfunction isResponderContext(ctx: EvaluationContext): ctx is ResponderEvaluationContext {\n\treturn (\n\t\t'responderOutput' in ctx &&\n\t\ttypeof (ctx as ResponderEvaluationContext).responderOutput === 'string' &&\n\t\t'responderEvals' in ctx &&\n\t\ttypeof (ctx as ResponderEvaluationContext).responderEvals === 'object'\n\t);\n}\n\nfunction parseJudgeResponse(content: string): ResponderJudgeResult {\n\t// Extract JSON from markdown code block if present\n\tconst jsonMatch = content.match(/```(?:json)?\\s*([\\s\\S]*?)```/);\n\tconst jsonStr = jsonMatch ? jsonMatch[1].trim() : content.trim();\n\ttry {\n\t\treturn JSON.parse(jsonStr) as ResponderJudgeResult;\n\t} catch {\n\t\tthrow new Error(`Failed to parse judge response as JSON: ${jsonStr.slice(0, 100)}...`);\n\t}\n}\n\nconst DIMENSION_KEYS = [\n\t'relevance',\n\t'accuracy',\n\t'completeness',\n\t'clarity',\n\t'tone',\n\t'criteriaMatch',\n\t'forbiddenPhrases',\n] as const;\n\nconst fb = (metric: string, score: number, kind: Feedback['kind'], comment?: string): Feedback => ({\n\tevaluator: EVALUATOR_NAME,\n\tmetric,\n\tscore,\n\tkind,","sourceCodeStart":45,"sourceCodeEnd":81,"githubUrl":"https://github.com/n8n-io/n8n/blob/5ac6606e81f67bb9534255570cd4e86fd8101eee/packages/@n8n/ai-workflow-builder.ee/evaluations/evaluators/responder/index.ts#L45-L81","documentation":"The responder evaluator asks the judge LLM for a JSON object matching `ResponderJudgeResult` (relevance/accuracy/completeness/clarity/tone/criteriaMatch/forbiddenPhrases dimensions + overallScore/summary). `parseJudgeResponse` first strips a markdown ```json fence if present, then `JSON.parse`s. If parsing fails, the first 100 chars of the judge's raw output are included in the thrown message to aid diagnosis.","triggerScenarios":"Judge model returns prose instead of JSON, returns JSON with trailing commas / comments / single quotes, wraps the object in extra text outside the code fence, or returns an empty string. The fence-stripping regex only matches a single ```json ... ``` block.","commonSituations":"Switching to a smaller/weaker judge model that doesn't follow output-format instructions; a prompt change that dropped the 'respond only with JSON' instruction; the model prepending an explanation like 'Here is the evaluation:' before the fence; rate-limit/truncation cutting the JSON mid-stream.","solutions":["Use `withStructuredOutput` / tool-calling for the judge (as the base evaluator chain does) so the model is forced to emit schema-conformant output instead of free-form JSON.","Tighten the judge system prompt to forbid any text outside the JSON/code fence, and switch to a model that follows those instructions reliably.","If you cannot change the evaluator, retry with `numJudges > 1` so a single malformed response is less likely to fail the whole run, and log the raw `content` for inspection."],"exampleFix":"// before\nfunction parseJudgeResponse(content: string): ResponderJudgeResult {\n  const jsonStr = ... ;\n  try { return JSON.parse(jsonStr) as ResponderJudgeResult; }\n  catch { throw new Error(`Failed to parse judge response as JSON: ${jsonStr.slice(0, 100)}...`); }\n}\n// after - tolerate prose-then-JSON and validate shape\nfunction parseJudgeResponse(content: string): ResponderJudgeResult {\n  const match = content.match(/\\{[\\s\\S]*\\}/);\n  const parsed = JSON.parse(match ? match[0] : content) as ResponderJudgeResult;\n  for (const k of DIMENSION_KEYS) if (!parsed[k]) throw new Error(`judge JSON missing dimension ${k}`);\n  return parsed;\n}","handlingStrategy":"try-catch","validationCode":"function looksLikeCompleteJson(s: string): boolean {\n  const t = s.trim();\n  if (!t.startsWith('{') || !t.endsWith('}')) return false;\n  try { JSON.parse(t); return true; } catch { return false; }\n}\n// only call parseJudgeResponse after this returns true; otherwise surface raw content for debugging","typeGuard":"function isResponderJudgeResult(v: unknown): v is ResponderJudgeResult {\n  if (!v || typeof v !== 'object') return false;\n  const r = v as Record<string, unknown>;\n  return ['relevance','accuracy','completeness','clarity','tone','criteriaMatch','forbiddenPhrases']\n    .every((k) => typeof r[k]?.score === 'number');\n}","tryCatchPattern":"try {\n  return parseJudgeResponse(content);\n} catch (e) {\n  if (e instanceof Error && e.message.startsWith('Failed to parse judge response as JSON')) {\n    logger.warn(`judge produced non-JSON output (first 100 chars): ${content.slice(0, 100)}`);\n    // fall back to a neutral score so one bad judge does not sink the example\n    return NEUTRAL_RESPONDER_RESULT;\n  }\n  throw e;\n}","preventionTips":["Prefer `withStructuredOutput(schema)` over free-form JSON when the judge supports it.","Pin a judge model known to follow output-format instructions and add a regression test that asserts parseability.","Log the raw judge `content` on failure so format regressions are diagnosable."],"tags":["llm","llm-judge","responder","evaluations","json","parsing"],"backgroundTag":null,"analyzedSha":"5ac6606e81f67bb9534255570cd4e86fd8101eee","analyzedAt":"2026-08-12T05:26:35.080Z","schemaVersion":2},"datasetVersion":"2026-08-13T14:17:21.547Z"}