{"record":{"id":"6489361f1022f4d9","repo":"santifer/career-ops","slug":"gemini-returned-an-invalid-career-ops-report-is","errorCode":null,"errorMessage":"Gemini returned an invalid career-ops report: ${issues.join('; ')}","messagePattern":"Gemini returned an invalid career-ops report: (.+?)","errorType":"validation","errorClass":null,"httpStatus":null,"severity":"error","filePath":"gemini-eval.mjs","lineNumber":285,"sourceCode":"  } else {\n    const summaryBlock = summary[1];\n    for (const key of ['COMPANY', 'ROLE', 'ARCHETYPE', 'LEGITIMACY']) {\n      const field = summaryBlock.match(new RegExp(`^\\\\s*${key}:\\\\s*(.+)$`, 'mi'));\n      const value = field?.[1]?.trim() ?? '';\n      if (!value || (key !== 'COMPANY' && value.toLowerCase() === 'unknown')) {\n        issues.push(`SCORE_SUMMARY ${key} is required`);\n      }\n    }\n\n    const score = summaryBlock.match(/^\\s*SCORE:\\s*([0-9]+(?:\\.[0-9]+)?)/mi);\n    const scoreValue = score ? Number(score[1]) : NaN;\n    if (!Number.isFinite(scoreValue) || scoreValue < 0 || scoreValue > 5) {\n      issues.push('SCORE_SUMMARY score must be a number between 0 and 5');\n    }\n  }\n\n  if (issues.length > 0) {\n    throw new Error(`Gemini returned an invalid career-ops report: ${issues.join('; ')}`);\n  }\n}\n\nfunction slugifyCompany(value) {\n  return String(value || '')\n    .toLowerCase()\n    .replace(/[^a-z0-9]+/g, '-')\n    .replace(/^-|-$/g, '') || 'unknown';\n}\n\n/**\n * Whether a value is a complete http(s) URL, and so can become a dedup key.\n * @param {string} value - Candidate posting URL.\n * @returns {boolean} True only for a parseable http/https URL with a host.\n */\nfunction isPostingUrl(value) {\n  try {\n    const parsed = new URL(value);","sourceCodeStart":267,"sourceCodeEnd":303,"githubUrl":"https://github.com/santifer/career-ops/blob/aac998c7ed7248ea853b720ceeb1fdbeb322fc5d/gemini-eval.mjs#L267-L303","documentation":"validateEvaluationShape in gemini-eval.mjs collects all structural problems found in the model's returned report (missing sections, non-numeric or out-of-range SCORE_SUMMARY score, etc.) and throws once with every issue joined by ';'. It guarantees the Gemini output conforms to the career-ops report contract before it is used downstream.","triggerScenarios":"The Gemini API response parsed but violated the expected report shape: a missing required section, SCORE_SUMMARY score absent/non-numeric/negative/above 5, or multiple rule violations accumulated in `issues`.","commonSituations":"Model version drift producing a re-formatted answer; a truncated response cutting off required sections; a prompt/temperature change making the model output prose instead of the structured template; a score given as '4/5' or 'good' instead of a number.","solutions":["Re-run the evaluation — non-deterministic model output often passes on retry.","Check the listed issue(s) and adjust the prompt/temperature to enforce the exact report template.","Pin a model version whose output format is known-good, and lower temperature for structured output.","Add explicit few-shot output-format instructions or a response schema if the API supports one."],"exampleFix":"// before\nconst score = '4/5'; // model output, rejected\n// after (prompt addition)\n\"SCORE_SUMMARY score MUST be a bare number between 0 and 5, e.g. `Score: 4.2`\"","handlingStrategy":"try-catch","validationCode":"function looksLikeReport(text) { return typeof text === 'string' && /SCORE_SUMMARY/i.test(text) && /Score:\\s*\\d+(\\.\\d+)?/.test(text); }","typeGuard":"const isValidScore = (v) => typeof v === 'number' && Number.isFinite(v) && v >= 0 && v <= 5;","tryCatchPattern":"try { validateEvaluationShape(report); } catch (e) { if (/invalid career-ops report/.test(e.message)) { retryWithReinforcedPrompt(); } else throw e; }","preventionTips":["Pin a known-good model version and low temperature for structured output.","Include the exact report template in the prompt with explicit numeric-score constraints.","Parse the model output defensively and re-prompt on failure rather than accepting malformed output.","Log the raw response alongside issues to spot format drift early."],"tags":["validation","llm","schema"],"backgroundTag":"unexpected-api-response-shape","analyzedSha":"aac998c7ed7248ea853b720ceeb1fdbeb322fc5d","analyzedAt":"2026-09-16T06:35:29.214Z","contentChangedAt":"2026-09-16T06:35:29.214Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}