{"record":{"id":"9410874c859510a0","repo":"santifer/career-ops","slug":"yourator-unexpected-api-response-on-page-page","errorCode":null,"errorMessage":"yourator: unexpected API response on page ${page} — expected { payload: { jobs: [...] } }, got keys: [${json ? Object.keys(json).join(', ') : 'null'}]","messagePattern":"yourator: unexpected API response on page (.+?) — expected (.+?) \\}, got keys: \\[(.+?)\\]","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"providers/yourator.mjs","lineNumber":189,"sourceCode":"/** @type {Provider} */\nexport default {\n  id: 'yourator',\n\n  async fetch(entry, ctx) {\n    assertYouratorUrl(FEED_BASE);\n    // ctx.maxPages is verify-portals.mjs's \"first page only\" health probe — it\n    // always wins over the entry's own bound.\n    const maxPages = Math.min(resolveMaxPages(entry), ctx?.maxPages ?? Number.POSITIVE_INFINITY);\n    const fallbackCompany = entry?.name;\n    const out = [];\n\n    for (let page = 1; page <= maxPages; page++) {\n      const url = `${FEED_BASE}?page=${page}`;\n      // redirect:'error' prevents SSRF via server-side redirects\n      const json = await ctx.fetchJson(url, { redirect: 'error' });\n      const jobs = json?.payload?.jobs;\n      if (!Array.isArray(jobs)) {\n        throw new Error(\n          `yourator: unexpected API response on page ${page} — expected { payload: { jobs: [...] } }, got keys: [${json ? Object.keys(json).join(', ') : 'null'}]`,\n        );\n      }\n      for (const j of jobs) {\n        const normalized = normalizeYouratorJob(j, fallbackCompany);\n        if (normalized) out.push(normalized);\n      }\n      // `hasMore` is the API's own end-of-board signal and the only stop\n      // condition: past the last page it answers with an empty array and\n      // hasMore:false. A short-page heuristic is deliberately NOT used — it\n      // cannot help (maxPages already bounds a runaway walk) and a single short\n      // intermediate page would silently truncate the board.\n      if (json.payload.hasMore !== true) break;\n      if (page < maxPages) {\n        await (ctx.sleep ? ctx.sleep(PAGE_DELAY_MS) : new Promise(r => setTimeout(r, PAGE_DELAY_MS)));\n      }\n    }\n    return out;","sourceCodeStart":171,"sourceCodeEnd":207,"githubUrl":"https://github.com/santifer/career-ops/blob/1696bec4d021768e7359f9aad6b329cba883da20/providers/yourator.mjs#L171-L207","documentation":"During fetch, each page of the Yourator feed is requested from `${FEED_BASE}?page=N` and the provider expects the documented envelope { payload: { jobs: [...] } }. If json.payload.jobs is missing or not an array, the upstream API contract has changed (or an error/HTML body was returned) and the provider throws rather than emitting garbage. The message includes the page number and the actual top-level keys received for diagnosis.","triggerScenarios":"ctx.fetchJson on page N returns JSON whose payload.jobs is absent or not an array: a v4→v5 API migration that renamed/reshaped payload; a rate-limit or maintenance JSON like { error: '...' }; a captive-portal/proxy returning JSON that isn't the jobs envelope; jobs: null or payload: null on an empty/edge page.","commonSituations":"Yourator ships a breaking API version change; a CDN/WAF intercepts and returns a JSON error body; transient upstream failure mid-walk on page >1; corporate proxy rewriting responses; following a stale FEED_BASE after a documented endpoint move.","solutions":["Check the reported keys: if you see error/message/rate-limit keys, retry the scan later or back off — it's a transient upstream response, not a schema change.","Verify the current API shape against the live endpoint https://www.yourator.co/api/v4/jobs?page=1 and update the provider's expected envelope (payload.jobs) if Yourator migrated versions.","If the schema genuinely changed, update normalizeYouratorJob's input path and the extraction in providers/yourator.mjs to the new field names, then rerun."],"exampleFix":"// before (provider expects v4 envelope)\nconst jobs = json?.payload?.jobs;\n\n// after (example: API moved to a top-level array)\nconst jobs = Array.isArray(json?.payload?.jobs) ? json.payload.jobs : Array.isArray(json?.jobs) ? json.jobs : null;","handlingStrategy":"try-catch","validationCode":"// Probe the feed shape before a full walk:\nconst probe = await fetch('https://www.yourator.co/api/v4/jobs?page=1').then(r => r.json());\nconst shapeOk = Array.isArray(probe?.payload?.jobs);\nif (!shapeOk) console.warn('Yourator API shape changed; check payload.jobs path', Object.keys(probe ?? {}));","typeGuard":"function isJobsEnvelope(json) {\n  return json != null\n    && typeof json === 'object'\n    && json.payload != null\n    && typeof json.payload === 'object'\n    && Array.isArray(json.payload.jobs);\n}","tryCatchPattern":"try {\n  await scanYouratorAllPages();\n} catch (e) {\n  if (e.message.startsWith('yourator: unexpected API response')) {\n    const page = e.message.match(/page (\\d+)/)?.[1];\n    console.error(`Yourator schema/availability issue on page ${page}; check for API version change or rate limiting, then retry.`);\n  } else throw e;\n}","preventionTips":["Pin/monitor the documented endpoint version (api/v4) and re-verify the envelope after any upstream announcement.","Rate-limit page walks and back off on 4xx/5xx so error JSON bodies are not mistaken for schema drift.","Add a smoke test asserting isJobsEnvelope on page 1 of the live feed in CI (or a recorded fixture).","Log the received top-level keys (the error already does) to distinguish rate-limit bodies from real schema changes."],"tags":["api","schema-change","http","json"],"backgroundTag":"unexpected-api-response-schema","analyzedSha":"1696bec4d021768e7359f9aad6b329cba883da20","analyzedAt":"2026-09-01T19:19:23.111Z","contentChangedAt":null,"schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}