{"record":{"id":"c4161c4da38249ba","repo":"santifer/career-ops","slug":"agentic-jobs-unexpected-api-response-shape-on-pag","errorCode":null,"errorMessage":"agentic-jobs: unexpected API response shape on page ${page} — \"data\" is missing or not an array","messagePattern":"agentic-jobs: unexpected API response shape on page (.+?) — \"data\" is missing or not an array","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"providers/agentic-jobs.mjs","lineNumber":171,"sourceCode":"  detect(entry) {\n    return entry?.provider === 'agentic-jobs' ? { url: SITE_ORIGIN } : null;\n  },\n\n  async fetch(_entry, ctx) {\n    const wait = (ms) => (ctx.sleep ? ctx.sleep(ms) : new Promise((r) => setTimeout(r, ms)));\n    const jobs = [];\n    const seen = new Set();\n    let total = null;\n\n    for (let page = 1; page <= MAX_PAGES; page++) {\n      if (page > 1) await wait(PAGE_DELAY_MS);\n      const url = assertAgenticUrl(`${API_BASE}/jobs?page=${page}`);\n      const json = await ctx.fetchJson(url, { redirect: 'error', headers: { accept: 'application/json' } });\n      // A missing/non-array `data` is a response-shape change, not a legitimate\n      // empty page (the API returns `data: []` for that) — fail loudly instead\n      // of silently truncating whatever pages were already collected.\n      if (!json || !Array.isArray(json.data)) {\n        throw new Error(`agentic-jobs: unexpected API response shape on page ${page} — \"data\" is missing or not an array`);\n      }\n      const records = json.data;\n      if (total === null) total = typeof json.meta?.total === 'number' ? json.meta.total : null;\n      // Trust the API's own reported page size over our constant, in case it\n      // ever differs from the documented default.\n      const effectivePageSize = typeof json.meta?.per_page === 'number' && json.meta.per_page > 0 ? json.meta.per_page : PAGE_SIZE;\n\n      for (const record of records) {\n        const job = normalizeAgenticJob(record);\n        if (job && !seen.has(job.url)) {\n          seen.add(job.url);\n          jobs.push(job);\n        }\n      }\n\n      if (jobs.length >= MAX_JOBS) break;\n      if (records.length < effectivePageSize) break; // short page — last one\n      if (total !== null && page * effectivePageSize >= total) break;","sourceCodeStart":153,"sourceCodeEnd":189,"githubUrl":"https://github.com/santifer/career-ops/blob/aac998c7ed7248ea853b720ceeb1fdbeb322fc5d/providers/agentic-jobs.mjs#L153-L189","documentation":"The agentic-jobs API is expected to return { data: [...jobs], meta: { total, per_page } }. If a page response has no `data` array, the contract has changed (or an error/HTML page came back as HTTP 200), and continuing would silently truncate the board to whatever pages were already collected. The provider throws loudly instead, treating it as a shape change rather than a legitimate empty page (which is data: []).","triggerScenarios":"ctx.fetchJson returns null, a JSON object without `data`, or `data` as a non-array for GET {API_BASE}/jobs?page=N — e.g. the API returned { error: ... } with HTTP 200, an HTML maintenance page parsed as JSON, or a versioned API change renaming/moving the data field.","commonSituations":"The remote API shipped a breaking schema change; a WAF/CDN intercepts the request and returns a JSON error body with 200; a proxy serves a captive-portal style JSON payload; running an old provider against a new API major version.","solutions":["Inspect the actual response: curl 'https://agentic-engineering-jobs.com/api/v1/jobs?page=1' and compare against the documented { data: [], meta: {} } shape.","Check the provider's OpenAPI spec (https://agentic-engineering-jobs.com/api/v1/openapi.json) for a renamed response field and update the parsing in providers/agentic-jobs.mjs accordingly.","Rule out an intercepting proxy/WAF by fetching directly and checking response headers (server, content-type) for the failing page.","Retry later if transient — a CDN error page with HTTP 200 will look exactly like this; a clean rerun that succeeds confirms it."],"exampleFix":"// defensive caller-side guard before relying on the run\nconst json = await ctx.fetchJson(url, { redirect: 'error', headers: { accept: 'application/json' } });\nif (!json || !Array.isArray(json.data)) {\n  // before: proceeding would silently truncate results\n  throw new Error(`agentic-jobs: bad page shape, keys=${json ? Object.keys(json) : 'null'}`);\n}","handlingStrategy":"type-guard","validationCode":"const res = await fetch('https://agentic-engineering-jobs.com/api/v1/jobs?page=1', { headers: { accept: 'application/json' } });\nconst json = await res.json();\nif (!json || !Array.isArray(json.data)) throw new Error('agentic-jobs API shape changed — check openapi.json before running scans');","typeGuard":"function hasAgenticPageShape(json) {\n  return !!json && typeof json === 'object' && Array.isArray(json.data) &&\n    (!json.meta || typeof json.meta === 'object');\n}","tryCatchPattern":"try {\n  jobs = await provider.fetch(entry, ctx);\n} catch (err) {\n  if (String(err.message).includes('unexpected API response shape')) {\n    // surface the raw body once for diagnosis\n    const raw = await fetch(`${API_BASE}/jobs?page=1`).then(r => r.text());\n    console.error('Raw first page:', raw.slice(0, 500));\n  }\n  throw err;\n}","preventionTips":["Before large sweeps, curl page 1 and confirm { data: [...], meta: {...} } still holds.","Watch the provider's OpenAPI spec for contract changes after upstream releases.","Don't catch-and-continue: a shape change silently truncates results, so failing loudly is correct."],"tags":["api","response-shape","schema-change","provider"],"backgroundTag":"unexpected-response-shape","analyzedSha":"aac998c7ed7248ea853b720ceeb1fdbeb322fc5d","analyzedAt":"2026-09-16T06:35:29.214Z","contentChangedAt":"2026-09-16T06:35:29.214Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}