santifer/career-ops · error · Error

agentic-jobs: unexpected API response shape on page ${page}

Error message

agentic-jobs: unexpected API response shape on page ${page} — "data" is missing or not an array

What it means

The agentic-jobs API is expected to return `{ data: [...], meta: {...} }` per page. After fetchJson, the provider checks json.data is an array; if json is null or json.data is missing/non-array, it throws naming the page. The comment is explicit: an empty page legitimately returns `data: []` (which passes this guard), so a missing/non-array `data` is specifically a response-shape change, not an empty result — fail loudly rather than silently truncating the pages already collected.

Source

Thrown at providers/agentic-jobs.mjs:171

  detect(entry) {
    return entry?.provider === 'agentic-jobs' ? { url: SITE_ORIGIN } : null;
  },

  async fetch(_entry, ctx) {
    const wait = (ms) => (ctx.sleep ? ctx.sleep(ms) : new Promise((r) => setTimeout(r, ms)));
    const jobs = [];
    const seen = new Set();
    let total = null;

    for (let page = 1; page <= MAX_PAGES; page++) {
      if (page > 1) await wait(PAGE_DELAY_MS);
      const url = assertAgenticUrl(`${API_BASE}/jobs?page=${page}`);
      const json = await ctx.fetchJson(url, { redirect: 'error', headers: { accept: 'application/json' } });
      // A missing/non-array `data` is a response-shape change, not a legitimate
      // empty page (the API returns `data: []` for that) — fail loudly instead
      // of silently truncating whatever pages were already collected.
      if (!json || !Array.isArray(json.data)) {
        throw new Error(`agentic-jobs: unexpected API response shape on page ${page} — "data" is missing or not an array`);
      }
      const records = json.data;
      if (total === null) total = typeof json.meta?.total === 'number' ? json.meta.total : null;
      // Trust the API's own reported page size over our constant, in case it
      // ever differs from the documented default.
      const effectivePageSize = typeof json.meta?.per_page === 'number' && json.meta.per_page > 0 ? json.meta.per_page : PAGE_SIZE;

      for (const record of records) {
        const job = normalizeAgenticJob(record);
        if (job && !seen.has(job.url)) {
          seen.add(job.url);
          jobs.push(job);
        }
      }

      if (jobs.length >= MAX_JOBS) break;
      if (records.length < effectivePageSize) break; // short page — last one
      if (total !== null && page * effectivePageSize >= total) break;

View on GitHub (pinned to 9b17a8ac97)

Solutions

  1. Read the error — it states page N and that 'data' is missing/not-array. Reproduce with `curl -H 'accept: application/json' '<API_BASE>/jobs?page=1'` to see the actual body.
  2. If upstream renamed `data`, update the check and normalizeAgenticJob in providers/agentic-jobs.mjs to the new key.
  3. If it is an error envelope (429/5xx body), the provider does NOT retry agentic-jobs (unlike a16z) — retry manually after the rate window (the API allows 30 req/60s; PAGE_DELAY_MS=2100ms stays under it).
  4. If fetchJson returned null, investigate the network/status content-type.

Example fix

// If upstream renamed `data` → `results`:
// before
if (!json || !Array.isArray(json.data)) { throw ... }
const records = json.data;

// after
if (!json || !Array.isArray(json.results)) { throw ... }
const records = json.results;
Defensive patterns

Strategy: try-catch

Validate before calling

// Probe page 1 and assert the documented shape before sweeping.
function isValidDataPage(json) {
  return !!json && Array.isArray(json.data);
}
const probe = await ctx.fetchJson(`${API_BASE}/jobs?page=1`, { redirect: 'error', headers: { accept: 'application/json' } });
if (!isValidDataPage(probe)) throw new Error('agentic-jobs feed shape unexpected; not sweeping.');

Type guard

/** @param {unknown} j @returns {j is { data: unknown[], meta?: { total?: number, per_page?: number } }} */
function isDataPage(j) {
  return !!j && typeof j === 'object' && Array.isArray((/** @type {any} */ (j)).data);
}

Try / catch

try {
  await provider.fetch(entry, ctx);
} catch (err) {
  if (/unexpected API response shape/.test(String(err?.message))) {
    console.error('agentic-jobs feed shape changed — curl the API and inspect `data`, then update the parser.');
  }
  throw err;
}

Prevention

When it happens

Trigger: The API returns a non-JSON body (null), an error envelope without data (e.g. {message:'...'}), renames `data` in a new API version, returns an HTML error page, or a CDN returns a wrapped object. An empty board page returns data:[] and does NOT trip this.

Common situations: Upstream API version bump renaming data; maintenance/error JSON envelope; rate-limit body that isn't the documented shape; the API endpoint moved and API_BASE points at a stale path.

Related errors

AI-assisted analysis of santifer/career-ops@9b17a8ac97 (2026-08-13). Data as JSON: /api/errors/c4161c4da38249ba. Report an issue: GitHub.