{"record":{"id":"a6b0bbdf3b9338c8","repo":"DIYgod/RSSHub","slug":"failed-to-extract-posts-from-the-next-js-rsc-paylo","errorCode":null,"errorMessage":"Failed to extract posts from the Next.js RSC payload","messagePattern":"Failed to extract posts from the Next\\.js RSC payload","errorType":"exception","errorClass":"Error","httpStatus":503,"severity":"error","filePath":"lib/routes/forwardfuture/originals.ts","lineNumber":102,"sourceCode":"    let match: RegExpExecArray | null;\n    let posts: OriginalPost[] = [];\n\n    while ((match = pushRegex.exec(html)) !== null) {\n        const payload = match[1];\n        if (!payload.includes('posts')) {\n            continue;\n        }\n\n        const decoded = JSON.parse(`\"${payload}\"`);\n        const extracted = extractPostsArray(decoded);\n        if (extracted) {\n            posts = extracted;\n            break;\n        }\n    }\n\n    if (posts.length === 0) {\n        throw new Error('Failed to extract posts from the Next.js RSC payload');\n    }\n\n    const items: DataItem[] = posts.slice(0, limit).map((post) => ({\n        title: post.title,\n        description: post.summary || undefined,\n        link: post.url,\n        pubDate: parseDate(post.dateUnix, 'X'),\n        author: post.authors.join(', '),\n        image: post.thumbnail,\n        category: post.categories.length > 0 ? post.categories : post.category === 'General' ? undefined : [post.category],\n    }));\n\n    return {\n        title: 'Forward Future - Originals',\n        link: 'https://forwardfuture.com/originals',\n        description: 'Original essays, columns, and analysis on AI from Forward Future contributors.',\n        item: items,\n        image: 'https://forwardfuture.com/images/logos/ff-icon.svg',","sourceCodeStart":84,"sourceCodeEnd":120,"githubUrl":"https://github.com/DIYgod/RSSHub/blob/bed535e0879dc71c5aff6f1e7bd1ac21ede40115/lib/routes/forwardfuture/originals.ts#L84-L120","documentation":"This route scrapes Forward Future's Next.js page and parses the React Server Components (RSC) streaming payload embedded in __next_f.push([1,\"...\"]) script chunks. It looks for a JSON array under a \"posts\" key and decodes the escaped string. If no chunk contains a parseable posts array, the route cannot recover any items and throws. This is a brittle scrape that breaks whenever the site's RSC payload shape or the page's data-loading changes.","triggerScenarios":"Forward Future ships a Next.js update that renames the 'posts' key, changes the RSC wire format, moves data out of the initial HTML into a client-side fetch, or the page returns an error/empty state (no published originals). The regex /__next_f\\.push\\(\\[1,\"...\"\\]\\)/g also fails if the push call signature changes (e.g. a different array element type).","commonSituations":"Next.js major version upgrades altering the RSC chunk format; the site switching from server-rendered posts to client-side data fetching; A/B tests or regional variants serving different markup; temporary empty state when no originals are published.","solutions":["Open https://forwardfuture.com/originals in a browser, view source, and confirm __next_f.push chunks still exist and contain 'posts'.","Update pushRegex if the push call signature changed (different array prefix, escaping).","Update extractPostsArray to handle the new location/key of the posts data, or switch to the site's actual data API if one exists in network requests.","Add a fallback that fetches the page via Playwright if the static HTML no longer contains the RSC payload."],"exampleFix":"// before\nif (posts.length === 0) {\n    throw new Error('Failed to extract posts from the Next.js RSC payload');\n}\n\n// after\nif (posts.length === 0) {\n    const sampleChunks = html.match(/__next_f\\.push\\(\\[[^\\]]*\\]\\)/g)?.slice(0, 3) ?? [];\n    throw new Error(`Failed to extract posts from the Next.js RSC payload. First chunks: ${JSON.stringify(sampleChunks)}`);\n}","handlingStrategy":"fallback","validationCode":"const html = await ofetch<string>('https://forwardfuture.com/originals');\nif (!html.includes('__next_f')) {\n  throw new Error('Page no longer contains Next.js RSC payload; site structure changed');\n}","typeGuard":"const hasRscPayload = (html: string): boolean => /__next_f\\.push/.test(html);","tryCatchPattern":null,"preventionTips":["Pin to a robust extraction that tolerates key renames (search multiple candidate keys).\nLog the first few push chunks on failure to diagnose format drift.\nMaintain a Playwright fallback for when static HTML no longer carries the payload."],"tags":["scraping","nextjs","rsc","fragile"],"backgroundTag":null,"analyzedSha":"bed535e0879dc71c5aff6f1e7bd1ac21ede40115","analyzedAt":"2026-08-12T19:29:35.364Z","schemaVersion":2},"datasetVersion":"2026-08-12T23:17:12.415Z"}