{"record":{"id":"28f2db7f7ffc90b9","repo":"santifer/career-ops","slug":"arbeitnow-unexpected-api-response-on-page-page","errorCode":null,"errorMessage":"arbeitnow: unexpected API response on page ${page} — expected { data: [...] }, got keys: [${json ? Object.keys(json).join(', ') : 'null'}]","messagePattern":"arbeitnow: unexpected API response on page (.+?) — expected (.+?), got keys: \\[(.+?)\\]","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"providers/arbeitnow.mjs","lineNumber":115,"sourceCode":"\n/** @type {Provider} */\nexport default {\n  id: 'arbeitnow',\n\n  async fetch(entry, ctx) {\n    assertArbeitnowUrl(FEED_BASE);\n    const maxPages = resolveMaxPages(entry);\n    const fallbackCompany = entry?.name;\n    const out = [];\n\n    for (let page = 1; page <= maxPages; page++) {\n      // Build the page URL directly (do NOT follow links.next — it carries a\n      // featured `?search=` term that would narrow the board).\n      const url = `${FEED_BASE}?page=${page}`;\n      // redirect:'error' prevents SSRF via server-side redirects\n      const json = await ctx.fetchJson(url, { redirect: 'error' });\n      if (!json || !Array.isArray(json.data)) {\n        throw new Error(\n          `arbeitnow: unexpected API response on page ${page} — expected { data: [...] }, got keys: [${json ? Object.keys(json).join(', ') : 'null'}]`,\n        );\n      }\n      for (const j of json.data) {\n        const normalized = normalizeArbeitnowJob(j, fallbackCompany);\n        if (normalized) out.push(normalized);\n      }\n      if (json.data.length < PER_PAGE) break; // short page → last page reached\n    }\n    return out;\n  },\n};\n","sourceCodeStart":97,"sourceCodeEnd":128,"githubUrl":"https://github.com/santifer/career-ops/blob/9b17a8ac97b398a496b38e423ae24e433b43254f/providers/arbeitnow.mjs#L97-L128","documentation":"arbeitnow's `fetch` expects each page JSON to be an object with a `data` array (the documented arbeitnow API shape). If the response is null or `data` is not an array, the provider throws rather than silently returning zero jobs — distinguishing an API contract break from a legitimately empty board.","triggerScenarios":"`ctx.fetchJson(url, { redirect:'error' })` returns null (non-JSON / empty body) or an object whose `data` field is missing or not an array. The message lists the actual top-level keys (or 'null') for diagnosis.","commonSituations":"The arbeitnow API changed its payload shape (renamed `data`, wrapped it), returned an HTML error page parsed as null, hit a CDN error JSON (`{error: ...}`), or the endpoint moved.","solutions":["Inspect the keys printed in the error to see what shape the API now returns.","Reproduce the request (`https://www.arbeitnow.com/api/job-board-api?page=1`) in a browser/curl and compare to the expected `{ data: [...] }`.","If the API renamed the array, update the `Array.isArray(json.data)` check and the `json.data.length < PER_PAGE` short-page logic.","If the response is null, check for a non-JSON reply (HTML maintenance page) and verify the fetch headers."],"exampleFix":"// before\nif (!json || !Array.isArray(json.data)) {\n  throw new Error(`arbeitnow: unexpected API response on page ${page} ...`);\n}\n\n// after — tolerate a documented wrapper while still failing loudly on garbage\nconst rows = Array.isArray(json?.data) ? json.data : Array.isArray(json?.jobs) ? json.jobs : null;\nif (!rows) {\n  throw new Error(`arbeitnow: unexpected API response on page ${page} — got keys: [${json ? Object.keys(json).join(', ') : 'null'}]`);\n}","handlingStrategy":"try-catch","validationCode":"// Probe one page at startup to confirm the API shape before the scan\nconst probe = await ctx.fetchJson(`${FEED_BASE}?page=1`, { redirect: 'error' });\nif (!probe || !Array.isArray(probe.data)) {\n  throw new Error(`arbeitnow: API contract drift — top-level keys: [${probe ? Object.keys(probe).join(', ') : 'null'}]`);\n}","typeGuard":"/** @param {any} j */\nfunction isArbeitnowFeed(j) {\n  return j !== null && typeof j === 'object' && Array.isArray(j.data);\n}","tryCatchPattern":"try {\n  const json = await ctx.fetchJson(url, { redirect: 'error' });\n  if (!isArbeitnowFeed(json)) throw new Error(`unexpected response — keys: [${json ? Object.keys(json).join(', ') : 'null'}]`);\n  // ...process json.data\n} catch (err) {\n  // log the page + keys, continue to next page or surface a contract-drift alert\n  console.error(`arbeitnow page ${page}: ${err.message}`);\n  throw err;\n}","preventionTips":["Add a startup contract probe so API drift fails fast with a clear message.","Pin the response shape with a type guard and surface actual keys on mismatch.","Watch for null responses (HTML maintenance pages) — they indicate a non-JSON reply, not contract drift."],"tags":["arbeitnow","api-contract","response-parsing","validation"],"backgroundTag":null,"analyzedSha":"9b17a8ac97b398a496b38e423ae24e433b43254f","analyzedAt":"2026-08-13T00:48:39.135Z","schemaVersion":2},"datasetVersion":"2026-08-13T04:17:16.726Z"}