{"record":{"id":"f37bfb8448fded49","repo":"santifer/career-ops","slug":"themuse-unexpected-api-response-on-page-page","errorCode":null,"errorMessage":"themuse: unexpected API response on page ${page} — expected { results: [...] }, got keys: [${json ? Object.keys(json).join(', ') : 'null'}]","messagePattern":"themuse: unexpected API response on page (.+?) — expected (.+?), got keys: \\[(.+?)\\]","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"warning","filePath":"providers/themuse.mjs","lineNumber":168,"sourceCode":"    }\n    const allResults = [...first.results];\n    const pageCount = Number.isInteger(first.page_count) && first.page_count > 1\n      ? Math.min(first.page_count, MAX_PAGES)\n      : 1;\n\n    // Pages 1+ stay tolerant: a page that exhausts retries, OR comes back\n    // with an unexpected shape (a successful fetch, no retry involved --\n    // caught here so it lands in the same truncation path instead of\n    // escaping uncaught and discarding allResults), truncates with a warning\n    // and returns whatever was already gathered instead of discarding it.\n    for (let page = 1; page < pageCount; page++) {\n      await sleep(INTER_PAGE_DELAY_MS, ctx);\n      const url = `${FEED_BASE}?page=${page}`;\n      let json;\n      try {\n        json = await fetchPageWithRetry(ctx, url, { redirect: 'error' });\n        if (!json || !Array.isArray(json.results)) {\n          throw new Error(\n            `themuse: unexpected API response on page ${page} — expected { results: [...] }, got keys: [${json ? Object.keys(json).join(', ') : 'null'}]`,\n          );\n        }\n      } catch (err) {\n        const attempts = Number.isInteger(err?.attempts) ? err.attempts : 1;\n        console.error(`⚠️  themuse: truncated at page ${page} of ${pageCount} after ${attempts} attempt${attempts === 1 ? '' : 's'} (${allResults.length} jobs gathered so far): ${err.message}`);\n        break;\n      }\n      allResults.push(...json.results);\n    }\n    return allResults.map(normalizeMuseJob).filter(Boolean);\n  },\n};\n","sourceCodeStart":150,"sourceCodeEnd":182,"githubUrl":"https://github.com/santifer/career-ops/blob/aac998c7ed7248ea853b720ceeb1fdbeb322fc5d/providers/themuse.mjs#L150-L182","documentation":"For pages after page 0, themuse's fetch loop tolerates failures: a page whose JSON lacks a results array throws this same shape error, but the surrounding catch logs a truncation warning ('⚠️ themuse: truncated at page N ...') and keeps the results accumulated so far instead of failing the whole scan. The error surfaces in logs as the reason a result set was cut short mid-pagination.","triggerScenarios":"During pagination of `${FEED_BASE}?page=${page}` (page >= 1), fetchPageWithRetry exhausts its attempts or returns JSON without an Array results field; the catch records the failed page and returns the partial allResults.","commonSituations":"Rate limiting kicking in after the first pages; Muse API intermittently 5xx-ing under load; MAX_PAGES cap combined with an unstable network; schema change landing mid-scan on later pages.","solutions":["Accept the partial results for this run and re-run the scan later — the provider is designed to degrade gracefully here.","Back off request rate (increase INTER_PAGE_DELAY_MS or reduce concurrent scans) if rate limiting caused the failures.","Inspect the failing page's body via curl to distinguish an error payload from a schema change.","If truncation is chronic, pin to fewer pages (lower MAX_PAGES) and schedule more frequent incremental scans.","If Muse changed its envelope, update the provider's results check."],"exampleFix":"// before\nconst url = `${FEED_BASE}?page=${page}`;\njson = await fetchPageWithRetry(ctx, url, { redirect: 'error' });\n// after — widen the inter-page delay when truncation recurs\nconst url = `${FEED_BASE}?page=${page}`;\nawait sleep(INTER_PAGE_DELAY_MS * 2, ctx);\njson = await fetchPageWithRetry(ctx, url, { redirect: 'error' });","handlingStrategy":"retry","validationCode":"function isMusePage(json) {\n  return !!json && typeof json === 'object' && Array.isArray(json.results);\n}\n// per page: if (!isMusePage(json)) back off and retry the page before giving up","typeGuard":"const hasResultsArray = (v) =>\n  v !== null && typeof v === 'object' && Array.isArray(v.results);","tryCatchPattern":"try {\n  const jobs = await themuseProvider.fetch(entry, ctx);\n  return jobs;\n} catch (err) {\n  // provider already logs truncation and returns partials; handle partials explicitly\n  if (String(err.message).includes('truncated at page')) {\n    console.warn('themuse scan truncated — schedule a retry with slower pacing');\n    return null;\n  }\n  throw err;\n}","preventionTips":["Increase inter-page delay or lower page counts when truncation warnings recur (rate limiting)","Run incremental scans more often instead of deep MAX_PAGES crawls","Monitor the '⚠️ themuse: truncated' log line as a rate/health metric","Persist already-fetched pages so a retried scan resumes instead of restarting"],"tags":["api","pagination","themuse","rate-limit","partial-results"],"backgroundTag":"unexpected-api-response-shape","analyzedSha":"aac998c7ed7248ea853b720ceeb1fdbeb322fc5d","analyzedAt":"2026-09-16T06:35:29.214Z","contentChangedAt":"2026-09-16T06:35:29.214Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}