{"record":{"id":"cbeb0a7c4681d0f1","repo":"santifer/career-ops","slug":"thehub-unexpected-api-response-on-page-page","errorCode":null,"errorMessage":"thehub: unexpected API response on page ${page} — expected { jobs: { docs: [...] } }, got keys: [${json ? Object.keys(json).join(', ') : 'null'}]","messagePattern":"thehub: unexpected API response on page (.+?) — expected (.+?) \\}, got keys: \\[(.+?)\\]","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"providers/thehub.mjs","lineNumber":144,"sourceCode":" *\n * @param {string} query the query string beyond `?`, e.g. `countryCode=EU` or `isRemote=true`\n * @param {number} maxPages\n * @param {string | undefined} fallbackCompany\n * @param {Map<string, {title: string, url: string, company: string, location: string}>} byUrl\n * @param {{ fetchJson: (url: string, opts?: object) => Promise<any> }} ctx\n * @param {{ succeededOnce: boolean }} state\n * @returns {Promise<boolean>}\n */\nasync function fetchScope(query, maxPages, fallbackCompany, byUrl, ctx, state) {\n  for (let page = 1; page <= maxPages; page++) {\n    const url = `${FEED_BASE}?page=${page}&${query}`;\n    let jobs;\n    try {\n      // redirect:'error' prevents SSRF via server-side redirects\n      const json = await ctx.fetchJson(url, { redirect: 'error' });\n      jobs = json && json.jobs;\n      if (!jobs || !Array.isArray(jobs.docs)) {\n        throw new Error(\n          `thehub: unexpected API response on page ${page} — expected { jobs: { docs: [...] } }, got keys: [${json ? Object.keys(json).join(', ') : 'null'}]`,\n        );\n      }\n    } catch (err) {\n      if (!state.succeededOnce) throw err;\n      console.error(`  ⚠ thehub: query \"${query}\" page ${page} failed (${err.message}) — keeping the ${byUrl.size} jobs collected so far`);\n      return false;\n    }\n    state.succeededOnce = true;\n    for (const j of jobs.docs) {\n      const normalized = normalizeHubJob(j, fallbackCompany);\n      if (normalized && !byUrl.has(normalized.url)) byUrl.set(normalized.url, normalized);\n    }\n    // Stop at the last page: a short page, or page >= the reported total pages.\n    if (jobs.docs.length < PER_PAGE) break;\n    if (Number.isInteger(jobs.pages) && page >= jobs.pages) break;\n  }\n  return true;","sourceCodeStart":126,"sourceCodeEnd":162,"githubUrl":"https://github.com/santifer/career-ops/blob/aac998c7ed7248ea853b720ceeb1fdbeb322fc5d/providers/thehub.mjs#L126-L162","documentation":"The thehub provider expects TheHub's API endpoint to return JSON shaped { jobs: { docs: [...] } }. When the response is null, a different shape, or jobs.docs is not an array, the provider throws. Inside fetchScope this throw is only fatal on the first successful query — once state.succeededOnce is true, later page failures degrade to a warning and the jobs collected so far are kept.","triggerScenarios":"fetchScope() calls ctx.fetchJson(url, { redirect: 'error' }) and the parsed body lacks json.jobs or json.jobs.docs is not an Array — e.g. an HTML error page parsed as text, a rate-limit JSON body like { message: '...' }, or an empty/null response on a given page.","commonSituations":"TheHub API returns a Cloudflare/challenge HTML page with 200; a query URL is malformed and the API answers with an error object; the API deployed a breaking schema change; heavy pagination hits a rate limit presenting a JSON error body.","solutions":["Re-run the scan — if it happened after page 0, the provider already kept partial results; a transient rate limit usually clears on retry.","Log the raw response body for the failing URL and check for an HTML challenge page or API error message.","Verify the constructed API URL (query params, page number) is correct for the current TheHub API.","Reduce request rate / page count if rate limiting is the cause.","If TheHub changed its schema, update the provider's response handling to the new shape."],"exampleFix":"// before\nconst json = await ctx.fetchJson(url, { redirect: 'error' });\n// after — inspect actual body when the shape is wrong\nconst json = await ctx.fetchJson(url, { redirect: 'error' });\nif (!json || !json.jobs || !Array.isArray(json.jobs.docs)) {\n  console.error('thehub raw body:', typeof json === 'string' ? json.slice(0, 300) : JSON.stringify(json).slice(0, 300));\n}","handlingStrategy":"try-catch","validationCode":"function isThehubJobsEnvelope(json) {\n  return !!json\n    && typeof json === 'object'\n    && json.jobs != null\n    && typeof json.jobs === 'object'\n    && Array.isArray(json.jobs.docs);\n}\n// check before consuming: if (!isThehubJobsEnvelope(json)) skip/warn","typeGuard":"const isJobsDocs = (json) =>\n  typeof json === 'object' && json !== null\n  && 'jobs' in json\n  && typeof json.jobs === 'object'\n  && Array.isArray(json.jobs?.docs);","tryCatchPattern":"try {\n  const json = await ctx.fetchJson(url, { redirect: 'error' });\n  if (!isJobsDocs(json)) throw new TypeError('thehub: unexpected envelope');\n  return json.jobs.docs;\n} catch (err) {\n  console.warn(`thehub page failed: ${err.message} — keeping partial results`);\n  return collectedSoFar;\n}","preventionTips":["Log the raw body (first ~300 chars) whenever the envelope check fails — it distinguishes HTML challenges from schema changes","Rate-limit scans against TheHub to avoid throttling error bodies served with 200","Snapshot a known-good API response as a fixture and diff periodically to catch schema drift","Trust partial results: keep jobs collected before the failure, as the provider does after succeededOnce"],"tags":["api","schema","thehub","unexpected-response"],"backgroundTag":"unexpected-api-response-shape","analyzedSha":"aac998c7ed7248ea853b720ceeb1fdbeb322fc5d","analyzedAt":"2026-09-16T06:35:29.214Z","contentChangedAt":"2026-09-16T06:35:29.214Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}