santifer/career-ops · error · Error

agentic-jobs: unexpected API response shape on page

Error message

agentic-jobs: unexpected API response shape on page ${page} — "data" is missing or not an array

What it means

The agentic-jobs API is expected to return { data: [...jobs], meta: { total, per_page } }. If a page response has no `data` array, the contract has changed (or an error/HTML page came back as HTTP 200), and continuing would silently truncate the board to whatever pages were already collected. The provider throws loudly instead, treating it as a shape change rather than a legitimate empty page (which is data: []).

Solutions

  1. Inspect the actual response: curl 'https://agentic-engineering-jobs.com/api/v1/jobs?page=1' and compare against the documented { data: [], meta: {} } shape.
  2. Check the provider's OpenAPI spec (https://agentic-engineering-jobs.com/api/v1/openapi.json) for a renamed response field and update the parsing in providers/agentic-jobs.mjs accordingly.
  3. Rule out an intercepting proxy/WAF by fetching directly and checking response headers (server, content-type) for the failing page.
  4. Retry later if transient — a CDN error page with HTTP 200 will look exactly like this; a clean rerun that succeeds confirms it.

Example fix

// defensive caller-side guard before relying on the run
const json = await ctx.fetchJson(url, { redirect: 'error', headers: { accept: 'application/json' } });
if (!json || !Array.isArray(json.data)) {
  // before: proceeding would silently truncate results
  throw new Error(`agentic-jobs: bad page shape, keys=${json ? Object.keys(json) : 'null'}`);
}
Defensive patterns

Strategy: type-guard

Validate before calling

const res = await fetch('https://agentic-engineering-jobs.com/api/v1/jobs?page=1', { headers: { accept: 'application/json' } });
const json = await res.json();
if (!json || !Array.isArray(json.data)) throw new Error('agentic-jobs API shape changed — check openapi.json before running scans');

Type guard

function hasAgenticPageShape(json) {
  return !!json && typeof json === 'object' && Array.isArray(json.data) &&
    (!json.meta || typeof json.meta === 'object');
}

Try / catch

try {
  jobs = await provider.fetch(entry, ctx);
} catch (err) {
  if (String(err.message).includes('unexpected API response shape')) {
    // surface the raw body once for diagnosis
    const raw = await fetch(`${API_BASE}/jobs?page=1`).then(r => r.text());
    console.error('Raw first page:', raw.slice(0, 500));
  }
  throw err;
}

Prevention

When it happens

Trigger: ctx.fetchJson returns null, a JSON object without `data`, or `data` as a non-array for GET {API_BASE}/jobs?page=N — e.g. the API returned { error: ... } with HTTP 200, an HTML maintenance page parsed as JSON, or a versioned API change renaming/moving the data field.

Common situations: The remote API shipped a breaking schema change; a WAF/CDN intercepts the request and returns a JSON error body with 200; a proxy serves a captive-portal style JSON payload; running an old provider against a new API major version.

Related errors

AI-assisted analysis of santifer/career-ops@aac998c7ed (2026-09-16). Data as JSON: /api/errors/c4161c4da38249ba. Report an issue: GitHub.

Appendix: source

Thrown at providers/agentic-jobs.mjs:171

  detect(entry) {
    return entry?.provider === 'agentic-jobs' ? { url: SITE_ORIGIN } : null;
  },

  async fetch(_entry, ctx) {
    const wait = (ms) => (ctx.sleep ? ctx.sleep(ms) : new Promise((r) => setTimeout(r, ms)));
    const jobs = [];
    const seen = new Set();
    let total = null;

    for (let page = 1; page <= MAX_PAGES; page++) {
      if (page > 1) await wait(PAGE_DELAY_MS);
      const url = assertAgenticUrl(`${API_BASE}/jobs?page=${page}`);
      const json = await ctx.fetchJson(url, { redirect: 'error', headers: { accept: 'application/json' } });
      // A missing/non-array `data` is a response-shape change, not a legitimate
      // empty page (the API returns `data: []` for that) — fail loudly instead
      // of silently truncating whatever pages were already collected.
      if (!json || !Array.isArray(json.data)) {
        throw new Error(`agentic-jobs: unexpected API response shape on page ${page} — "data" is missing or not an array`);
      }
      const records = json.data;
      if (total === null) total = typeof json.meta?.total === 'number' ? json.meta.total : null;
      // Trust the API's own reported page size over our constant, in case it
      // ever differs from the documented default.
      const effectivePageSize = typeof json.meta?.per_page === 'number' && json.meta.per_page > 0 ? json.meta.per_page : PAGE_SIZE;

      for (const record of records) {
        const job = normalizeAgenticJob(record);
        if (job && !seen.has(job.url)) {
          seen.add(job.url);
          jobs.push(job);
        }
      }

      if (jobs.length >= MAX_JOBS) break;
      if (records.length < effectivePageSize) break; // short page — last one
      if (total !== null && page * effectivePageSize >= total) break;

View on GitHub (pinned to aac998c7ed)