{"record":{"id":"49bec4804fc3093f","repo":"santifer/career-ops","slug":"4dayweek-unexpected-api-response-on-page-page","errorCode":null,"errorMessage":"4dayweek: unexpected API response on page ${page} — expected { jobs: [...] }, got keys: [${json ? Object.keys(json).join(', ') : 'null'}]","messagePattern":"4dayweek: unexpected API response on page (.+?) — expected (.+?), got keys: \\[(.+?)\\]","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"providers/4dayweek.mjs","lineNumber":145,"sourceCode":"\n/** @type {Provider} */\nexport default {\n  id: '4dayweek',\n\n  detect: detectFourDayEntry,\n\n  async fetch(entry, ctx) {\n    assertFourDayUrl(FEED_BASE);\n    const maxPages = resolveMaxPages(entry);\n    const fallbackCompany = entry?.name;\n    const out = [];\n\n    for (let page = 1; page <= maxPages; page++) {\n      const url = `${FEED_BASE}?page=${page}`;\n      // redirect:'error' prevents SSRF via server-side redirects\n      const json = await ctx.fetchJson(url, { redirect: 'error' });\n      if (!json || !Array.isArray(json.jobs)) {\n        throw new Error(\n          `4dayweek: unexpected API response on page ${page} — expected { jobs: [...] }, got keys: [${json ? Object.keys(json).join(', ') : 'null'}]`,\n        );\n      }\n      for (const j of json.jobs) {\n        const normalized = normalize4dwJob(j, fallbackCompany);\n        if (normalized) out.push(normalized);\n      }\n      if (json.has_more === false) break; // last page per the API flag\n      if (json.jobs.length < PER_PAGE) break; // short page → last page\n    }\n    return out;\n  },\n};\n","sourceCodeStart":127,"sourceCodeEnd":159,"githubUrl":"https://github.com/santifer/career-ops/blob/1696bec4d021768e7359f9aad6b329cba883da20/providers/4dayweek.mjs#L127-L159","documentation":"In the 4dayweek provider's paginated fetch loop, each page's JSON from ctx.fetchJson() must contain a jobs array. When the response is null/non-object or lacks jobs (e.g. the API returned an error envelope, HTML, or a rate-limit body), the provider throws with the page number and the actual top-level keys so you can see what came back instead.","triggerScenarios":"fetching https://4dayweek.io feed with ?page=N where the API returns an unexpected shape: an error object {error: ...}, a rate-limit/throttle body, an empty page beyond the last (some APIs return {} instead of {jobs: []}), a Cloudflare HTML interstitial that fetchJson parsed loosely, or an API schema change.","commonSituations":"Hitting past the last page while resolveMaxPages() allowed more pages; the 4dayweek API being temporarily down or changed its response schema; a WAF/CDN blocking the client with a non-Job JSON/HTML body; network middleware returning a JSON error envelope.","solutions":["Log the full response body for the failing page (the message lists the keys) to identify what the API actually returned.","Retry the failing page — transient blocks/rate limits often present as malformed bodies; add backoff.","Lower max_pages on the provider entry so pagination stops at the real last page instead of over-fetching into error pages.","Check whether 4dayweek changed its API shape and update normalize4dwJob/expectations accordingly.","If a CDN/WAF is intercepting, send a proper User-Agent or fetch from an allowed network."],"exampleFix":"// before (over-fetching past the end)\n// { \"max_pages\": 20 } while API only has 3 pages -> page 4 returns {}\n// after\n// { \"max_pages\": 3 }  or handle empty last page:\nif (!json || !Array.isArray(json.jobs)) {\n  if (json && Object.keys(json).length === 0) break; // empty page = end of results\n  throw new Error(`4dayweek: unexpected API response on page ${page}`);\n}","handlingStrategy":"retry","validationCode":"function looksLikeJobsPage(json) {\n  return json !== null && typeof json === 'object' && Array.isArray(json.jobs);\n}\n// after fetchJson:\nif (!looksLikeJobsPage(json)) {\n  if (json && Object.keys(json).length === 0) return out; // empty last page\n  throw new Error(`4dayweek page ${page}: bad shape, keys=${Object.keys(json ?? {})}`);\n}","typeGuard":"function isJobsPage(v) {\n  return typeof v === 'object' && v !== null && Array.isArray(v.jobs);\n}","tryCatchPattern":"try {\n  json = await ctx.fetchJson(url, { redirect: 'error' });\n} catch (err) {\n  if (isJobsPage(err)) throw err;\n  if (attempt < 3) { await sleep(2 ** attempt * 1000); continue; } // backoff retry\n  throw err;\n}","preventionTips":["Set max_pages no higher than the real page count so pagination never over-fetches into error pages.","Treat an empty object on the first page of a batch as end-of-results rather than an error.","Add exponential backoff on malformed pages — they're often transient WAF/rate-limit bodies.","Monitor for 4dayweek API schema changes with a periodic smoke test asserting { jobs: [] } shape."],"tags":["api","network","response-validation","pagination"],"backgroundTag":"unexpected-api-response-shape","analyzedSha":"1696bec4d021768e7359f9aad6b329cba883da20","analyzedAt":"2026-09-01T19:19:23.111Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}