DIYgod/RSSHub · error · Error

Failed to extract posts from the Next.js RSC payload

Error message

Failed to extract posts from the Next.js RSC payload

What it means

This route scrapes Forward Future's Next.js page and parses the React Server Components (RSC) streaming payload embedded in __next_f.push([1,"..."]) script chunks. It looks for a JSON array under a "posts" key and decodes the escaped string. If no chunk contains a parseable posts array, the route cannot recover any items and throws. This is a brittle scrape that breaks whenever the site's RSC payload shape or the page's data-loading changes.

Source

Thrown at lib/routes/forwardfuture/originals.ts:102

    let match: RegExpExecArray | null;
    let posts: OriginalPost[] = [];

    while ((match = pushRegex.exec(html)) !== null) {
        const payload = match[1];
        if (!payload.includes('posts')) {
            continue;
        }

        const decoded = JSON.parse(`"${payload}"`);
        const extracted = extractPostsArray(decoded);
        if (extracted) {
            posts = extracted;
            break;
        }
    }

    if (posts.length === 0) {
        throw new Error('Failed to extract posts from the Next.js RSC payload');
    }

    const items: DataItem[] = posts.slice(0, limit).map((post) => ({
        title: post.title,
        description: post.summary || undefined,
        link: post.url,
        pubDate: parseDate(post.dateUnix, 'X'),
        author: post.authors.join(', '),
        image: post.thumbnail,
        category: post.categories.length > 0 ? post.categories : post.category === 'General' ? undefined : [post.category],
    }));

    return {
        title: 'Forward Future - Originals',
        link: 'https://forwardfuture.com/originals',
        description: 'Original essays, columns, and analysis on AI from Forward Future contributors.',
        item: items,
        image: 'https://forwardfuture.com/images/logos/ff-icon.svg',

View on GitHub (pinned to bed535e087)

Solutions

1. Open https://forwardfuture.com/originals in a browser, view source, and confirm __next_f.push chunks still exist and contain 'posts'.
2. Update pushRegex if the push call signature changed (different array prefix, escaping).
3. Update extractPostsArray to handle the new location/key of the posts data, or switch to the site's actual data API if one exists in network requests.
4. Add a fallback that fetches the page via Playwright if the static HTML no longer contains the RSC payload.

Example fix

// before
if (posts.length === 0) {
    throw new Error('Failed to extract posts from the Next.js RSC payload');
}

// after
if (posts.length === 0) {
    const sampleChunks = html.match(/__next_f\.push\(\[[^\]]*\]\)/g)?.slice(0, 3) ?? [];
    throw new Error(`Failed to extract posts from the Next.js RSC payload. First chunks: ${JSON.stringify(sampleChunks)}`);
}

const html = await ofetch<string>('https://forwardfuture.com/originals');
if (!html.includes('__next_f')) {
  throw new Error('Page no longer contains Next.js RSC payload; site structure changed');
}

const hasRscPayload = (html: string): boolean => /__next_f\.push/.test(html);

When it happens

Trigger: Forward Future ships a Next.js update that renames the 'posts' key, changes the RSC wire format, moves data out of the initial HTML into a client-side fetch, or the page returns an error/empty state (no published originals). The regex /__next_f\.push\(\[1,"..."\]\)/g also fails if the push call signature changes (e.g. a different array element type).

Common situations: Next.js major version upgrades altering the RSC chunk format; the site switching from server-rendered posts to client-side data fetching; A/B tests or regional variants serving different markup; temporary empty state when no originals are published.

Related errors

• Failed to parse blogList from RSC data
• 无法解析页面数据，请检查漫画 ID 是否正确或页面结构是否变动
• 无法解析页面 Props 数据
• 无法解析页面 HTML 数据，可能触发了反爬策略或页面结构巨变
• failed to parse AJAX response

AI-assisted analysis of DIYgod/RSSHub@bed535e087 (2026-08-12). Data as JSON: /api/errors/a6b0bbdf3b9338c8. Report an issue: GitHub.