{"record":{"id":"7e2e99341d471b5c","repo":"santifer/career-ops","slug":"join-next-data-not-found-or-unexpected-struct","errorCode":null,"errorMessage":"join: __NEXT_DATA__ not found or unexpected structure","messagePattern":"join: __NEXT_DATA__ not found or unexpected structure","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"providers/join.mjs","lineNumber":62,"sourceCode":"\n  detect(entry) {\n    return extractSlug(entry.careers_url) ? { url: entry.careers_url } : null;\n  },\n\n  async fetch(entry, ctx) {\n    const slug = extractSlug(entry.careers_url);\n    if (!slug) throw new Error('join: cannot extract slug from careers_url');\n\n    const baseUrl = `https://join.com/companies/${slug}`;\n    const allItems = [];\n\n    // redirect:'error' prevents SSRF via server-side redirects; baseUrl is\n    // always reconstructed as https://join.com/... so the host is pinned\n    // regardless of the original careers_url.\n    const firstHtml = await ctx.fetchText(baseUrl, { redirect: 'error' });\n    const firstData = extractNextData(firstHtml);\n    const state = firstData?.props?.pageProps?.initialState;\n    if (!state) throw new Error('join: __NEXT_DATA__ not found or unexpected structure');\n\n    const firstJobs = state.jobs?.items;\n    if (!Array.isArray(firstJobs)) throw new Error('join: __NEXT_DATA__ not found or unexpected structure');\n    allItems.push(...firstJobs);\n\n    // Honor a context page cap — verify-portals' liveness probe sets\n    // `ctx.maxPages: 1` so it only needs to know a board is live, not its\n    // full count (mirrors providers/workday.mjs). No effect on real scans,\n    // which don't set ctx.maxPages. Kept separate from `maxPages` below so\n    // the \"raise max_pages\" warning only fires when the entry-level cap is\n    // what actually truncated the board, not the health-check probe.\n    const ctxMaxPages = Number(ctx?.maxPages);\n    const ctxCap = ctxMaxPages > 0 ? ctxMaxPages : Infinity;\n    const reportedPageCount = state.jobs?.pagination?.pageCount || 0;\n    const maxPages = resolveMaxPages(entry);\n    const pageCount = Math.min(reportedPageCount, maxPages, ctxCap);\n    for (let page = 2; page <= pageCount; page++) {\n      const html = await ctx.fetchText(`${baseUrl}?page=${page}`, { redirect: 'error' });","sourceCodeStart":44,"sourceCodeEnd":80,"githubUrl":"https://github.com/santifer/career-ops/blob/aac998c7ed7248ea853b720ceeb1fdbeb322fc5d/providers/join.mjs#L44-L80","documentation":"join.com renders jobs server-side with Next.js; the provider parses the embedded <script id=\"__NEXT_DATA__\"> JSON via extractNextData() and reads props.pageProps.initialState. It throws this error when the script tag is missing entirely, the JSON fails to parse, or initialState is absent — i.e. the response is not a valid join.com company board page.","triggerScenarios":"ctx.fetchText(baseUrl, {redirect:'error'}) returned HTML without __NEXT_DATA__ (e.g. a Cloudflare/human-check page, a 404 page for a dead slug, a maintenance page), or the JSON parsed but initialState is null/missing.","commonSituations":"Bot-protection interstitial served instead of the page; company closed its join.com board; join.com deployed a framework change (e.g. moving off Next.js or renaming initialState) breaking the extractor.","solutions":["Fetch the board URL manually and confirm it is a real careers page; if it shows a bot check or 404, the slug is wrong or the board is gone — fix careers_url or remove the entry.","Retry later if a transient bot-protection or CDN issue served non-board HTML.","If join.com changed its page structure, update extractNextData()/the initialState path in providers/join.mjs to match the new __NEXT_DATA__ layout."],"exampleFix":"// before (portals.yml)\ncareers_url: https://join.com/companies/acme-gone\n// after\ncareers_url: https://join.com/companies/acme","handlingStrategy":"try-catch","validationCode":"const html = await fetch(boardUrl).then(r => r.text());\nif (!html.includes('__NEXT_DATA__')) console.warn(`${boardUrl} did not serve a Next.js board page`);","typeGuard":null,"tryCatchPattern":"try {\n  await provider.fetch(entry, ctx);\n} catch (e) {\n  if (e.message.includes('__NEXT_DATA__ not found')) {\n    // bot-check page, dead slug, or join.com markup change — verify board manually\n  }\n}","preventionTips":["Verify the board opens in a plain browser (no interstitial) before adding it to portals.yml.","Remove entries for companies that left join.com promptly.","Pin a smoke test that scans one known-good join.com board to catch framework changes early."],"tags":["scraping","nextjs","html-parsing","join-com"],"backgroundTag":"unexpected-response-shape","analyzedSha":"aac998c7ed7248ea853b720ceeb1fdbeb322fc5d","analyzedAt":"2026-09-16T06:35:29.214Z","contentChangedAt":"2026-09-16T06:35:29.214Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}