santifer/career-ops · warning · Error

themuse: unexpected API response on page

Error message

themuse: unexpected API response on page ${page} — expected { results: [...] }, got keys: [${json ? Object.keys(json).join(', ') : 'null'}]

What it means

For pages after page 0, themuse's fetch loop tolerates failures: a page whose JSON lacks a results array throws this same shape error, but the surrounding catch logs a truncation warning ('⚠️ themuse: truncated at page N ...') and keeps the results accumulated so far instead of failing the whole scan. The error surfaces in logs as the reason a result set was cut short mid-pagination.

Solutions

  1. Accept the partial results for this run and re-run the scan later — the provider is designed to degrade gracefully here.
  2. Back off request rate (increase INTER_PAGE_DELAY_MS or reduce concurrent scans) if rate limiting caused the failures.
  3. Inspect the failing page's body via curl to distinguish an error payload from a schema change.
  4. If truncation is chronic, pin to fewer pages (lower MAX_PAGES) and schedule more frequent incremental scans.
  5. If Muse changed its envelope, update the provider's results check.

Example fix

// before
const url = `${FEED_BASE}?page=${page}`;
json = await fetchPageWithRetry(ctx, url, { redirect: 'error' });
// after — widen the inter-page delay when truncation recurs
const url = `${FEED_BASE}?page=${page}`;
await sleep(INTER_PAGE_DELAY_MS * 2, ctx);
json = await fetchPageWithRetry(ctx, url, { redirect: 'error' });
Defensive patterns

Strategy: retry

Validate before calling

function isMusePage(json) {
  return !!json && typeof json === 'object' && Array.isArray(json.results);
}
// per page: if (!isMusePage(json)) back off and retry the page before giving up

Type guard

const hasResultsArray = (v) =>
  v !== null && typeof v === 'object' && Array.isArray(v.results);

Try / catch

try {
  const jobs = await themuseProvider.fetch(entry, ctx);
  return jobs;
} catch (err) {
  // provider already logs truncation and returns partials; handle partials explicitly
  if (String(err.message).includes('truncated at page')) {
    console.warn('themuse scan truncated — schedule a retry with slower pacing');
    return null;
  }
  throw err;
}

Prevention

When it happens

Trigger: During pagination of `${FEED_BASE}?page=${page}` (page >= 1), fetchPageWithRetry exhausts its attempts or returns JSON without an Array results field; the catch records the failed page and returns the partial allResults.

Common situations: Rate limiting kicking in after the first pages; Muse API intermittently 5xx-ing under load; MAX_PAGES cap combined with an unstable network; schema change landing mid-scan on later pages.

Understand the failure class

Background: "invalid response format", "malformed payload", "missing data field": when an API returns 200 but the response shape is wrong — this error's family across 23 libraries.

Related errors

AI-assisted analysis of santifer/career-ops@aac998c7ed (2026-09-16). Data as JSON: /api/errors/f37bfb8448fded49. Report an issue: GitHub.

Appendix: source

Thrown at providers/themuse.mjs:168

    }
    const allResults = [...first.results];
    const pageCount = Number.isInteger(first.page_count) && first.page_count > 1
      ? Math.min(first.page_count, MAX_PAGES)
      : 1;

    // Pages 1+ stay tolerant: a page that exhausts retries, OR comes back
    // with an unexpected shape (a successful fetch, no retry involved --
    // caught here so it lands in the same truncation path instead of
    // escaping uncaught and discarding allResults), truncates with a warning
    // and returns whatever was already gathered instead of discarding it.
    for (let page = 1; page < pageCount; page++) {
      await sleep(INTER_PAGE_DELAY_MS, ctx);
      const url = `${FEED_BASE}?page=${page}`;
      let json;
      try {
        json = await fetchPageWithRetry(ctx, url, { redirect: 'error' });
        if (!json || !Array.isArray(json.results)) {
          throw new Error(
            `themuse: unexpected API response on page ${page} — expected { results: [...] }, got keys: [${json ? Object.keys(json).join(', ') : 'null'}]`,
          );
        }
      } catch (err) {
        const attempts = Number.isInteger(err?.attempts) ? err.attempts : 1;
        console.error(`⚠️  themuse: truncated at page ${page} of ${pageCount} after ${attempts} attempt${attempts === 1 ? '' : 's'} (${allResults.length} jobs gathered so far): ${err.message}`);
        break;
      }
      allResults.push(...json.results);
    }
    return allResults.map(normalizeMuseJob).filter(Boolean);
  },
};

View on GitHub (pinned to aac998c7ed)