{"record":{"id":"28f2db7f7ffc90b9","repo":"santifer/career-ops","slug":"arbeitnow-unexpected-api-response-on-page-page","errorCode":null,"errorMessage":"arbeitnow: unexpected API response on page ${page} — expected { data: [...] }, got keys: [${json ? Object.keys(json).join(', ') : 'null'}]","messagePattern":"arbeitnow: unexpected API response on page (.+?) — expected (.+?), got keys: \\[(.+?)\\]","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"providers/arbeitnow.mjs","lineNumber":115,"sourceCode":"\n/** @type {Provider} */\nexport default {\n  id: 'arbeitnow',\n\n  async fetch(entry, ctx) {\n    assertArbeitnowUrl(FEED_BASE);\n    const maxPages = resolveMaxPages(entry);\n    const fallbackCompany = entry?.name;\n    const out = [];\n\n    for (let page = 1; page <= maxPages; page++) {\n      // Build the page URL directly (do NOT follow links.next — it carries a\n      // featured `?search=` term that would narrow the board).\n      const url = `${FEED_BASE}?page=${page}`;\n      // redirect:'error' prevents SSRF via server-side redirects\n      const json = await ctx.fetchJson(url, { redirect: 'error' });\n      if (!json || !Array.isArray(json.data)) {\n        throw new Error(\n          `arbeitnow: unexpected API response on page ${page} — expected { data: [...] }, got keys: [${json ? Object.keys(json).join(', ') : 'null'}]`,\n        );\n      }\n      for (const j of json.data) {\n        const normalized = normalizeArbeitnowJob(j, fallbackCompany);\n        if (normalized) out.push(normalized);\n      }\n      if (json.data.length < PER_PAGE) break; // short page → last page reached\n    }\n    return out;\n  },\n};\n","sourceCodeStart":97,"sourceCodeEnd":128,"githubUrl":"https://github.com/santifer/career-ops/blob/aac998c7ed7248ea853b720ceeb1fdbeb322fc5d/providers/arbeitnow.mjs#L97-L128","documentation":"The arbeitnow feed must return { data: [...] } per page. If the response is null, not an object, or `data` is missing/not an array, the provider throws — including the actual top-level keys it saw — because the documented contract changed or a non-API payload (error page, HTML) came back. It treats this as a shape change, not an empty board.","triggerScenarios":"GET https://www.arbeitnow.com/api/job-board-api?page=N returns anything without an array `data` — a CDN/WAF HTML block page, a JSON error envelope ({ message, errors }), rate-limit JSON, or an API version bump renaming data.","commonSituations":"Rate limiting or bot protection from arbeitnow serving an interstitial during aggressive sweeps; the API changed its envelope; a captive network/proxy intercepts TLS and returns its own page; transient 5xx body parsed as JSON.","solutions":["Read the got-keys list in the message: an HTML block page means the string starts with '<' — check for WAF/rate limiting and slow down or add delays between page requests.","If keys show an error envelope, read its message (e.g. throttle or auth) and adjust (backoff, API key).","Diff the live response (curl https://www.arbeitnow.com/api/job-board-api?page=1) against the documented { data, links, meta } shape and update the parser if the contract changed.","Retry later if transient — a single bad page mid-run is fatal here, so confirm with a manual rerun before treating it as a code issue."],"exampleFix":"// before: generic retry that hammers the WAF\nfor (const page of pages) await fetchPage(page);\n// after: pace requests under rate limits\nfor (const page of pages) {\n  await fetchPage(page);\n  await new Promise(r => setTimeout(r, 1500));\n}","handlingStrategy":"type-guard","validationCode":"const res = await fetch('https://www.arbeitnow.com/api/job-board-api?page=1');\nconst ct = res.headers.get('content-type') || '';\nif (!ct.includes('application/json')) throw new Error(`arbeitnow returned ${ct} — likely WAF/rate-limit page`);\nconst json = await res.json();\nif (!Array.isArray(json?.data)) throw new Error(`unexpected envelope, keys: ${Object.keys(json || {})}`);","typeGuard":"function hasArbeitnowPageShape(json) {\n  return !!json && typeof json === 'object' && Array.isArray(json.data);\n}","tryCatchPattern":"try {\n  jobs = await provider.fetch(entry, ctx);\n} catch (err) {\n  if (String(err.message).includes('unexpected API response')) {\n    console.error('Envelope check the got-keys list in the message — HTML means bot protection: back off and slow page pacing');\n  }\n  throw err;\n}","preventionTips":["Pace page requests (delays between pages) to avoid triggering arbeitnow's rate limiting.","Check content-type before JSON.parse so HTML block pages are recognized immediately.","Before long sweeps, probe page 1 to confirm the { data, links, meta } envelope."],"tags":["api","response-shape","rate-limit","provider"],"backgroundTag":"unexpected-response-shape","analyzedSha":"aac998c7ed7248ea853b720ceeb1fdbeb322fc5d","analyzedAt":"2026-09-16T06:35:29.214Z","contentChangedAt":"2026-09-16T06:35:29.214Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}