{"record":{"id":"7e2e99341d471b5c","repo":"santifer/career-ops","slug":"join-next-data-not-found-or-unexpected-struct","errorCode":null,"errorMessage":"join: __NEXT_DATA__ not found or unexpected structure","messagePattern":"join: __NEXT_DATA__ not found or unexpected structure","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"providers/join.mjs","lineNumber":62,"sourceCode":"\n  detect(entry) {\n    return extractSlug(entry.careers_url) ? { url: entry.careers_url } : null;\n  },\n\n  async fetch(entry, ctx) {\n    const slug = extractSlug(entry.careers_url);\n    if (!slug) throw new Error('join: cannot extract slug from careers_url');\n\n    const baseUrl = `https://join.com/companies/${slug}`;\n    const allItems = [];\n\n    // redirect:'error' prevents SSRF via server-side redirects; baseUrl is\n    // always reconstructed as https://join.com/... so the host is pinned\n    // regardless of the original careers_url.\n    const firstHtml = await ctx.fetchText(baseUrl, { redirect: 'error' });\n    const firstData = extractNextData(firstHtml);\n    const state = firstData?.props?.pageProps?.initialState;\n    if (!state) throw new Error('join: __NEXT_DATA__ not found or unexpected structure');\n\n    const firstJobs = state.jobs?.items;\n    if (!Array.isArray(firstJobs)) throw new Error('join: __NEXT_DATA__ not found or unexpected structure');\n    allItems.push(...firstJobs);\n\n    // Honor a context page cap — verify-portals' liveness probe sets\n    // `ctx.maxPages: 1` so it only needs to know a board is live, not its\n    // full count (mirrors providers/workday.mjs). No effect on real scans,\n    // which don't set ctx.maxPages. Kept separate from `maxPages` below so\n    // the \"raise max_pages\" warning only fires when the entry-level cap is\n    // what actually truncated the board, not the health-check probe.\n    const ctxMaxPages = Number(ctx?.maxPages);\n    const ctxCap = ctxMaxPages > 0 ? ctxMaxPages : Infinity;\n    const reportedPageCount = state.jobs?.pagination?.pageCount || 0;\n    const maxPages = resolveMaxPages(entry);\n    const pageCount = Math.min(reportedPageCount, maxPages, ctxCap);\n    for (let page = 2; page <= pageCount; page++) {\n      const html = await ctx.fetchText(`${baseUrl}?page=${page}`, { redirect: 'error' });","sourceCodeStart":44,"sourceCodeEnd":80,"githubUrl":"https://github.com/santifer/career-ops/blob/9b17a8ac97b398a496b38e423ae24e433b43254f/providers/join.mjs#L44-L80","documentation":"Thrown by join.fetch() on the first page when extractNextData(firstHtml) returns null OR the parsed data lacks props.pageProps.initialState. extractNextData looks for a <script> tag containing __NEXT_DATA__ and JSON.parses its content; it returns null if the tag is absent, the HTML isn't a string, or the JSON is malformed. This is a fail-closed guard: a missing/unexpected Next.js data shape is treated as a scraper break, not an empty board, so it is logged rather than silently reporting zero jobs.","triggerScenarios":"join.com changed its SSR structure (renamed pageProps or initialState keys); the request returned an error page (403/429/500 HTML) instead of the company board; a WAF or bot challenge (Cloudflare interstitial) replaced the real page; the company slug no longer exists and join.com serves a generic page.","commonSituations":"join.com shipped a frontend change that renamed or restructured the __NEXT_DATA__ payload; rate-limiting returned an HTML challenge page; the company migrated off join.com.","solutions":["Open https://join.com/companies/<slug> in a browser and confirm the board still renders with a __NEXT_DATA__ script tag.","If you are being rate-limited, reduce scan frequency or add a delay between requests.","If join.com changed its data shape, update extractNextData / the initialState access path in providers/join.mjs."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":"import { extractNextData } from './providers/join.mjs';\n// Probe the board shape before a full scan:\nasync function joinBoardShapeOk(fetchText, baseUrl) {\n  const html = await fetchText(baseUrl, { redirect: 'error' });\n  const state = extractNextData(html)?.props?.pageProps?.initialState;\n  return Boolean(state);\n}","typeGuard":"/** @param {any} data @returns {boolean} */\nfunction hasJoinInitialState(data) {\n  return Boolean(data && data?.props?.pageProps?.initialState);\n}","tryCatchPattern":"try {\n  jobs = await provider.fetch(entry, ctx);\n} catch (err) {\n  if (/__NEXT_DATA__ not found/.test(err.message)) {\n    console.error(`join ${entry.name}: SSR shape changed or blocked — investigate manually`);\n  }\n  throw err;\n}","preventionTips":["Treat __NEXT_DATA__ scrapers as fragile — any frontend deploy can break them.","Monitor scan logs for 'structure changed' patterns and surface them as provider health alerts.","Prefer official APIs over SSR scraping where the board offers one."],"tags":["join","nextjs","scraping","structure-change","fail-closed"],"backgroundTag":null,"analyzedSha":"9b17a8ac97b398a496b38e423ae24e433b43254f","analyzedAt":"2026-08-13T00:48:39.135Z","schemaVersion":2},"datasetVersion":"2026-08-13T04:17:16.726Z"}