{"record":{"id":"c4161c4da38249ba","repo":"santifer/career-ops","slug":"agentic-jobs-unexpected-api-response-shape-on-pag","errorCode":null,"errorMessage":"agentic-jobs: unexpected API response shape on page ${page} — \"data\" is missing or not an array","messagePattern":"agentic-jobs: unexpected API response shape on page (.+?) — \"data\" is missing or not an array","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"providers/agentic-jobs.mjs","lineNumber":171,"sourceCode":"  detect(entry) {\n    return entry?.provider === 'agentic-jobs' ? { url: SITE_ORIGIN } : null;\n  },\n\n  async fetch(_entry, ctx) {\n    const wait = (ms) => (ctx.sleep ? ctx.sleep(ms) : new Promise((r) => setTimeout(r, ms)));\n    const jobs = [];\n    const seen = new Set();\n    let total = null;\n\n    for (let page = 1; page <= MAX_PAGES; page++) {\n      if (page > 1) await wait(PAGE_DELAY_MS);\n      const url = assertAgenticUrl(`${API_BASE}/jobs?page=${page}`);\n      const json = await ctx.fetchJson(url, { redirect: 'error', headers: { accept: 'application/json' } });\n      // A missing/non-array `data` is a response-shape change, not a legitimate\n      // empty page (the API returns `data: []` for that) — fail loudly instead\n      // of silently truncating whatever pages were already collected.\n      if (!json || !Array.isArray(json.data)) {\n        throw new Error(`agentic-jobs: unexpected API response shape on page ${page} — \"data\" is missing or not an array`);\n      }\n      const records = json.data;\n      if (total === null) total = typeof json.meta?.total === 'number' ? json.meta.total : null;\n      // Trust the API's own reported page size over our constant, in case it\n      // ever differs from the documented default.\n      const effectivePageSize = typeof json.meta?.per_page === 'number' && json.meta.per_page > 0 ? json.meta.per_page : PAGE_SIZE;\n\n      for (const record of records) {\n        const job = normalizeAgenticJob(record);\n        if (job && !seen.has(job.url)) {\n          seen.add(job.url);\n          jobs.push(job);\n        }\n      }\n\n      if (jobs.length >= MAX_JOBS) break;\n      if (records.length < effectivePageSize) break; // short page — last one\n      if (total !== null && page * effectivePageSize >= total) break;","sourceCodeStart":153,"sourceCodeEnd":189,"githubUrl":"https://github.com/santifer/career-ops/blob/9b17a8ac97b398a496b38e423ae24e433b43254f/providers/agentic-jobs.mjs#L153-L189","documentation":"The agentic-jobs API is expected to return `{ data: [...], meta: {...} }` per page. After fetchJson, the provider checks json.data is an array; if json is null or json.data is missing/non-array, it throws naming the page. The comment is explicit: an empty page legitimately returns `data: []` (which passes this guard), so a missing/non-array `data` is specifically a response-shape change, not an empty result — fail loudly rather than silently truncating the pages already collected.","triggerScenarios":"The API returns a non-JSON body (null), an error envelope without data (e.g. {message:'...'}), renames `data` in a new API version, returns an HTML error page, or a CDN returns a wrapped object. An empty board page returns data:[] and does NOT trip this.","commonSituations":"Upstream API version bump renaming data; maintenance/error JSON envelope; rate-limit body that isn't the documented shape; the API endpoint moved and API_BASE points at a stale path.","solutions":["Read the error — it states page N and that 'data' is missing/not-array. Reproduce with `curl -H 'accept: application/json' '<API_BASE>/jobs?page=1'` to see the actual body.","If upstream renamed `data`, update the check and normalizeAgenticJob in providers/agentic-jobs.mjs to the new key.","If it is an error envelope (429/5xx body), the provider does NOT retry agentic-jobs (unlike a16z) — retry manually after the rate window (the API allows 30 req/60s; PAGE_DELAY_MS=2100ms stays under it).","If fetchJson returned null, investigate the network/status content-type."],"exampleFix":"// If upstream renamed `data` → `results`:\n// before\nif (!json || !Array.isArray(json.data)) { throw ... }\nconst records = json.data;\n\n// after\nif (!json || !Array.isArray(json.results)) { throw ... }\nconst records = json.results;","handlingStrategy":"try-catch","validationCode":"// Probe page 1 and assert the documented shape before sweeping.\nfunction isValidDataPage(json) {\n  return !!json && Array.isArray(json.data);\n}\nconst probe = await ctx.fetchJson(`${API_BASE}/jobs?page=1`, { redirect: 'error', headers: { accept: 'application/json' } });\nif (!isValidDataPage(probe)) throw new Error('agentic-jobs feed shape unexpected; not sweeping.');","typeGuard":"/** @param {unknown} j @returns {j is { data: unknown[], meta?: { total?: number, per_page?: number } }} */\nfunction isDataPage(j) {\n  return !!j && typeof j === 'object' && Array.isArray((/** @type {any} */ (j)).data);\n}","tryCatchPattern":"try {\n  await provider.fetch(entry, ctx);\n} catch (err) {\n  if (/unexpected API response shape/.test(String(err?.message))) {\n    console.error('agentic-jobs feed shape changed — curl the API and inspect `data`, then update the parser.');\n  }\n  throw err;\n}","preventionTips":["Pin a feed fixture in CI; a diff catches upstream `data` renames.","Remember this provider does NOT auto-retry transient errors (unlike a16z) — if 429s are likely, add a bounded retry around fetchJson.","Distinguish 'empty page' (data:[] passes) from 'shape change' (this guard) in diagnostics."],"tags":["agentic-jobs","api","response-shape","provider","validation"],"backgroundTag":null,"analyzedSha":"9b17a8ac97b398a496b38e423ae24e433b43254f","analyzedAt":"2026-08-13T00:48:39.135Z","schemaVersion":2},"datasetVersion":"2026-08-13T04:17:16.726Z"}