santifer/career-ops · error · Error

thehub: unexpected API response on page

Error message

thehub: unexpected API response on page ${page} — expected { jobs: { docs: [...] } }, got keys: [${json ? Object.keys(json).join(', ') : 'null'}]

What it means

The thehub provider expects TheHub's API endpoint to return JSON shaped { jobs: { docs: [...] } }. When the response is null, a different shape, or jobs.docs is not an array, the provider throws. Inside fetchScope this throw is only fatal on the first successful query — once state.succeededOnce is true, later page failures degrade to a warning and the jobs collected so far are kept.

Solutions

Re-run the scan — if it happened after page 0, the provider already kept partial results; a transient rate limit usually clears on retry.
Log the raw response body for the failing URL and check for an HTML challenge page or API error message.
Verify the constructed API URL (query params, page number) is correct for the current TheHub API.
Reduce request rate / page count if rate limiting is the cause.
If TheHub changed its schema, update the provider's response handling to the new shape.

Example fix

// before
const json = await ctx.fetchJson(url, { redirect: 'error' });
// after — inspect actual body when the shape is wrong
const json = await ctx.fetchJson(url, { redirect: 'error' });
if (!json || !json.jobs || !Array.isArray(json.jobs.docs)) {
  console.error('thehub raw body:', typeof json === 'string' ? json.slice(0, 300) : JSON.stringify(json).slice(0, 300));
}

Defensive patterns

Strategy: try-catch

Validate before calling

function isThehubJobsEnvelope(json) {
  return !!json
    && typeof json === 'object'
    && json.jobs != null
    && typeof json.jobs === 'object'
    && Array.isArray(json.jobs.docs);
}
// check before consuming: if (!isThehubJobsEnvelope(json)) skip/warn

Type guard

const isJobsDocs = (json) =>
  typeof json === 'object' && json !== null
  && 'jobs' in json
  && typeof json.jobs === 'object'
  && Array.isArray(json.jobs?.docs);

Try / catch

try {
  const json = await ctx.fetchJson(url, { redirect: 'error' });
  if (!isJobsDocs(json)) throw new TypeError('thehub: unexpected envelope');
  return json.jobs.docs;
} catch (err) {
  console.warn(`thehub page failed: ${err.message} — keeping partial results`);
  return collectedSoFar;
}

Prevention

Log the raw body (first ~300 chars) whenever the envelope check fails — it distinguishes HTML challenges from schema changes
Rate-limit scans against TheHub to avoid throttling error bodies served with 200
Snapshot a known-good API response as a fixture and diff periodically to catch schema drift
Trust partial results: keep jobs collected before the failure, as the provider does after succeededOnce

When it happens

Trigger: fetchScope() calls ctx.fetchJson(url, { redirect: 'error' }) and the parsed body lacks json.jobs or json.jobs.docs is not an Array — e.g. an HTML error page parsed as text, a rate-limit JSON body like { message: '...' }, or an empty/null response on a given page.

Common situations: TheHub API returns a Cloudflare/challenge HTML page with 200; a query URL is malformed and the API answers with an error object; the API deployed a breaking schema change; heavy pagination hits a rate limit presenting a JSON error body.

Understand the failure class

Background: "invalid response format", "malformed payload", "missing data field": when an API returns 200 but the response shape is wrong — this error's family across 23 libraries.

Related errors

AI-assisted analysis of santifer/career-ops@aac998c7ed (2026-09-16). Data as JSON: /api/errors/cbeb0a7c4681d0f1. Report an issue: GitHub.

Appendix: source

Thrown at providers/thehub.mjs:144

 *
 * @param {string} query the query string beyond `?`, e.g. `countryCode=EU` or `isRemote=true`
 * @param {number} maxPages
 * @param {string | undefined} fallbackCompany
 * @param {Map<string, {title: string, url: string, company: string, location: string}>} byUrl
 * @param {{ fetchJson: (url: string, opts?: object) => Promise<any> }} ctx
 * @param {{ succeededOnce: boolean }} state
 * @returns {Promise<boolean>}
 */
async function fetchScope(query, maxPages, fallbackCompany, byUrl, ctx, state) {
  for (let page = 1; page <= maxPages; page++) {
    const url = `${FEED_BASE}?page=${page}&${query}`;
    let jobs;
    try {
      // redirect:'error' prevents SSRF via server-side redirects
      const json = await ctx.fetchJson(url, { redirect: 'error' });
      jobs = json && json.jobs;
      if (!jobs || !Array.isArray(jobs.docs)) {
        throw new Error(
          `thehub: unexpected API response on page ${page} — expected { jobs: { docs: [...] } }, got keys: [${json ? Object.keys(json).join(', ') : 'null'}]`,
        );
      }
    } catch (err) {
      if (!state.succeededOnce) throw err;
      console.error(`  ⚠ thehub: query "${query}" page ${page} failed (${err.message}) — keeping the ${byUrl.size} jobs collected so far`);
      return false;
    }
    state.succeededOnce = true;
    for (const j of jobs.docs) {
      const normalized = normalizeHubJob(j, fallbackCompany);
      if (normalized && !byUrl.has(normalized.url)) byUrl.set(normalized.url, normalized);
    }
    // Stop at the last page: a short page, or page >= the reported total pages.
    if (jobs.docs.length < PER_PAGE) break;
    if (Number.isInteger(jobs.pages) && page >= jobs.pages) break;
  }
  return true;

View on GitHub (pinned to aac998c7ed)