{"record":{"id":"6489361f1022f4d9","repo":"santifer/career-ops","slug":"gemini-returned-an-invalid-career-ops-report-is","errorCode":null,"errorMessage":"Gemini returned an invalid career-ops report: ${issues.join('; ')}","messagePattern":"Gemini returned an invalid career-ops report: (.+?)","errorType":"validation","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"gemini-eval.mjs","lineNumber":209,"sourceCode":"  } else {\n    const summaryBlock = summary[1];\n    for (const key of ['COMPANY', 'ROLE', 'ARCHETYPE', 'LEGITIMACY']) {\n      const field = summaryBlock.match(new RegExp(`^\\\\s*${key}:\\\\s*(.+)$`, 'mi'));\n      const value = field?.[1]?.trim() ?? '';\n      if (!value || (key !== 'COMPANY' && value.toLowerCase() === 'unknown')) {\n        issues.push(`SCORE_SUMMARY ${key} is required`);\n      }\n    }\n\n    const score = summaryBlock.match(/^\\s*SCORE:\\s*([0-9]+(?:\\.[0-9]+)?)/mi);\n    const scoreValue = score ? Number(score[1]) : NaN;\n    if (!Number.isFinite(scoreValue) || scoreValue < 0 || scoreValue > 5) {\n      issues.push('SCORE_SUMMARY score must be a number between 0 and 5');\n    }\n  }\n\n  if (issues.length > 0) {\n    throw new Error(`Gemini returned an invalid career-ops report: ${issues.join('; ')}`);\n  }\n}\n\nfunction slugifyCompany(value) {\n  return String(value || '')\n    .toLowerCase()\n    .replace(/[^a-z0-9]+/g, '-')\n    .replace(/^-|-$/g, '') || 'unknown';\n}\n\nfunction tsvSafe(value) {\n  return String(value ?? '').replace(/[\\t\\r\\n]+/g, ' ').trim();\n}\n\nfunction normalizedTrackerScore(value) {\n  const clean = tsvSafe(value);\n  if (!clean || clean === '?') return 'N/A';\n  return /\\/5$/i.test(clean) ? clean : `${clean}/5`;","sourceCodeStart":191,"sourceCodeEnd":227,"githubUrl":"https://github.com/santifer/career-ops/blob/9b17a8ac97b398a496b38e423ae24e433b43254f/gemini-eval.mjs#L191-L227","documentation":"validateEvaluationShape checks that Gemini's raw text output matches the career-ops report contract: headers for Blocks A through G must be present, a SCORE_SUMMARY block delimited by ---SCORE_SUMMARY--- / ---END_SUMMARY--- must exist, it must contain COMPANY, ROLE, ARCHETYPE, and LEGITIMACY fields (ROLE/ARCHETYPE/LEGITIMACY cannot be 'unknown'), and SCORE must be a finite number in 0–5. All failures are collected and reported in one message.","triggerScenarios":"Gemini truncated output mid-report (missing later blocks); omitted SCORE_SUMMARY delimiters; wrote 'unknown' for ROLE or ARCHETYPE; SCORE was outside 0-5 or non-numeric; model restructured the output format; token limit cut off the response.","commonSituations":"Model output instability across runs; the prompt template was changed and no longer requests a required block; a very long JD pushed the response over the output token limit; model returned a refusal or error instead of the report.","solutions":["Re-run the evaluation — model output can vary between calls.","Inspect the raw Gemini output to see which block or field is missing or malformed.","If output is truncated, reduce the input size (shorten the JD or cv.md context) or increase the model output token limit.","Ensure the prompt template explicitly requests all Blocks A–G and the SCORE_SUMMARY block."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// validateEvaluationShape is called internally; callers wrap the Gemini call\ntry {\n  const reportText = await callGemini(prompt);\n  validateEvaluationShape(reportText);\n  // proceed to write report\n} catch (err) {\n  if (err.message.includes('invalid career-ops report')) {\n    console.error('Gemini output failed validation:', err.message);\n    // Retry, or log raw output for inspection\n    process.exit(1);\n  }\n  throw err;\n}","preventionTips":["Re-run evaluations on validation failure — model output varies.","Keep prompts under the model's output token limit to avoid truncation.","Ensure the prompt template explicitly requests Blocks A–G and the SCORE_SUMMARY block.","Log raw model output alongside parsed reports for debugging."],"tags":["llm","validation","gemini","eval","report"],"backgroundTag":null,"analyzedSha":"9b17a8ac97b398a496b38e423ae24e433b43254f","analyzedAt":"2026-08-13T00:48:39.135Z","schemaVersion":2},"datasetVersion":"2026-08-13T04:17:16.726Z"}