DIYgod/RSSHub · error · Error

无法解析页面 HTML 数据，可能触发了反爬策略或页面结构巨变

Error message

无法解析页面 HTML 数据，可能触发了反爬策略或页面结构巨变

What it means

This error fires when the comic-walker route fetches a manga detail page but the Next.js __NEXT_DATA__ script tag is empty or absent. Comic Walker (KADOKAWA) uses Next.js with React Query dehydration, embedding all API data in this tag. Without it, the route has no way to extract manga metadata or episode lists.

Source

Thrown at lib/routes/comic-walker/manga.ts:52

    handler: async (ctx) => {
        const { id } = ctx.req.param();
        const baseUrl = 'https://comic-walker.com';

        const fetchUrl = `${baseUrl}/detail/${id}?episodeType=first`;
        const openUrl = `${baseUrl}/detail/${id}`;

        const response = await ofetch<string>(fetchUrl, {
            headers: {
                'Accept-Language': 'ja,en-US;q=0.9,en;q=0.8',
            },
        });

        const $ = load(response);
        const nextDataText = $('#__NEXT_DATA__').text();

        if (!nextDataText) {
            throw new Error('无法解析页面 HTML 数据，可能触发了反爬策略或页面结构巨变');
        }

        const nextData = JSON.parse(nextDataText);
        const queries = nextData.props?.pageProps?.dehydratedState?.queries || [];

        const workQuery = queries.find((q: any) => q.queryKey?.includes('/api/contents/details/work') || (Array.isArray(q.queryKey) && q.queryKey.some((k: any) => typeof k === 'string' && k.includes('work'))));

        if (!workQuery || !workQuery.state?.data) {
            throw new Error('无法在 HTML 缓存中提取核心数据对象');
        }

        const data = workQuery.state.data;
        const work = data.work;

        if (!work) {
            throw new Error('成功获取数据对象，但未找到作品基本信息');
        }

View on GitHub (pinned to bed535e087)

Solutions

1. Check if the URL is accessible from a normal browser — if it shows a challenge page, the route needs anti-bot handling.
2. Add config.trueUA and a more complete browser-like header set to the ofetch call.
3. Reduce request frequency by ensuring cache.tryGet is used properly upstream.
4. If the page loads in a browser but not via ofetch, consider switching to Puppeteer with request interception.
5. Verify the fetchUrl is still correct — the API/data endpoint may have changed.

Example fix

// before
const response = await ofetch<string>(fetchUrl, {
    headers: {
        'Accept-Language': 'ja,en-US;q=0.9,en;q=0.8',
    },
});

// after — add full browser headers
const response = await ofetch<string>(fetchUrl, {
    headers: {
        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
        'Accept-Language': 'ja,en-US;q=0.9,en;q=0.8',
        'User-Agent': config.trueUA,
    },
});

// Validate response contains expected markup before parsing
if (typeof response !== 'string' || response.length < 500) {
    throw new Error('Response too short or not HTML — possible block or error page');
}
if (!response.includes('__NEXT_DATA__')) {
    throw new Error('Page missing __NEXT_DATA__ — possible anti-scraping block');
}

function hasNextDataScript(html: string): boolean {
    return html.includes('id="__NEXT_DATA__"');
}

// Retry with backoff for transient anti-bot blocks
let attempt = 0;
let nextDataText = '';
while (attempt < 2) {
    const response = await ofetch<string>(fetchUrl, { headers: getBrowserHeaders() });
    const $ = load(response);
    nextDataText = $('#__NEXT_DATA__').text();
    if (nextDataText) break;
    attempt++;
}
if (!nextDataText) throw new Error('Failed to get __NEXT_DATA__ after retries');

When it happens

Trigger: The ofetch request to the comic-walker fetchUrl returns HTML where $('#__NEXT_DATA__').text() is falsy. This can result from anti-scraping middleware (comic-walker is known for bot detection), a CDN-level block returning a minimal error page, or a site-wide frontend rewrite.

Common situations: Frequent automated requests trigger rate-limiting or Cloudflare challenges; the Accept-Language header alone is insufficient and a full browser-like header set is needed; the site temporarily goes into maintenance mode serving a static page without Next.js data.

Related errors

• 无法解析页面数据，请检查漫画 ID 是否正确或页面结构是否变动
• 无法解析页面 Props 数据
• 无法在 HTML 缓存中提取核心数据对象
• 成功获取数据对象，但未找到作品基本信息
• HTML 缓存中无章节!

AI-assisted analysis of DIYgod/RSSHub@bed535e087 (2026-08-12). Data as JSON: /api/errors/2b86a23c030795ee. Report an issue: GitHub.