{"record":{"id":"2b86a23c030795ee","repo":"DIYgod/RSSHub","slug":"html","errorCode":null,"errorMessage":"无法解析页面 HTML 数据，可能触发了反爬策略或页面结构巨变","messagePattern":"无法解析页面 HTML 数据，可能触发了反爬策略或页面结构巨变","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"lib/routes/comic-walker/manga.ts","lineNumber":52,"sourceCode":"\n    handler: async (ctx) => {\n        const { id } = ctx.req.param();\n        const baseUrl = 'https://comic-walker.com';\n\n        const fetchUrl = `${baseUrl}/detail/${id}?episodeType=first`;\n        const openUrl = `${baseUrl}/detail/${id}`;\n\n        const response = await ofetch<string>(fetchUrl, {\n            headers: {\n                'Accept-Language': 'ja,en-US;q=0.9,en;q=0.8',\n            },\n        });\n\n        const $ = load(response);\n        const nextDataText = $('#__NEXT_DATA__').text();\n\n        if (!nextDataText) {\n            throw new Error('无法解析页面 HTML 数据，可能触发了反爬策略或页面结构巨变');\n        }\n\n        const nextData = JSON.parse(nextDataText);\n        const queries = nextData.props?.pageProps?.dehydratedState?.queries || [];\n\n        const workQuery = queries.find((q: any) => q.queryKey?.includes('/api/contents/details/work') || (Array.isArray(q.queryKey) && q.queryKey.some((k: any) => typeof k === 'string' && k.includes('work'))));\n\n        if (!workQuery || !workQuery.state?.data) {\n            throw new Error('无法在 HTML 缓存中提取核心数据对象');\n        }\n\n        const data = workQuery.state.data;\n        const work = data.work;\n\n        if (!work) {\n            throw new Error('成功获取数据对象，但未找到作品基本信息');\n        }\n","sourceCodeStart":34,"sourceCodeEnd":70,"githubUrl":"https://github.com/DIYgod/RSSHub/blob/bed535e0879dc71c5aff6f1e7bd1ac21ede40115/lib/routes/comic-walker/manga.ts#L34-L70","documentation":"This error fires when the comic-walker route fetches a manga detail page but the Next.js __NEXT_DATA__ script tag is empty or absent. Comic Walker (KADOKAWA) uses Next.js with React Query dehydration, embedding all API data in this tag. Without it, the route has no way to extract manga metadata or episode lists.","triggerScenarios":"The ofetch request to the comic-walker fetchUrl returns HTML where $('#__NEXT_DATA__').text() is falsy. This can result from anti-scraping middleware (comic-walker is known for bot detection), a CDN-level block returning a minimal error page, or a site-wide frontend rewrite.","commonSituations":"Frequent automated requests trigger rate-limiting or Cloudflare challenges; the Accept-Language header alone is insufficient and a full browser-like header set is needed; the site temporarily goes into maintenance mode serving a static page without Next.js data.","solutions":["Check if the URL is accessible from a normal browser — if it shows a challenge page, the route needs anti-bot handling.","Add config.trueUA and a more complete browser-like header set to the ofetch call.","Reduce request frequency by ensuring cache.tryGet is used properly upstream.","If the page loads in a browser but not via ofetch, consider switching to Puppeteer with request interception.","Verify the fetchUrl is still correct — the API/data endpoint may have changed."],"exampleFix":"// before\nconst response = await ofetch<string>(fetchUrl, {\n    headers: {\n        'Accept-Language': 'ja,en-US;q=0.9,en;q=0.8',\n    },\n});\n\n// after — add full browser headers\nconst response = await ofetch<string>(fetchUrl, {\n    headers: {\n        'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',\n        'Accept-Language': 'ja,en-US;q=0.9,en;q=0.8',\n        'User-Agent': config.trueUA,\n    },\n});","handlingStrategy":"retry","validationCode":"// Validate response contains expected markup before parsing\nif (typeof response !== 'string' || response.length < 500) {\n    throw new Error('Response too short or not HTML — possible block or error page');\n}\nif (!response.includes('__NEXT_DATA__')) {\n    throw new Error('Page missing __NEXT_DATA__ — possible anti-scraping block');\n}","typeGuard":"function hasNextDataScript(html: string): boolean {\n    return html.includes('id=\"__NEXT_DATA__\"');\n}","tryCatchPattern":"// Retry with backoff for transient anti-bot blocks\nlet attempt = 0;\nlet nextDataText = '';\nwhile (attempt < 2) {\n    const response = await ofetch<string>(fetchUrl, { headers: getBrowserHeaders() });\n    const $ = load(response);\n    nextDataText = $('#__NEXT_DATA__').text();\n    if (nextDataText) break;\n    attempt++;\n}\nif (!nextDataText) throw new Error('Failed to get __NEXT_DATA__ after retries');","preventionTips":["Send complete browser-like headers (Accept, Accept-Language, User-Agent) when scraping comic-walker.","Cache aggressively to reduce request frequency.","Consider using Puppeteer for sites with aggressive anti-bot protection.","Monitor response sizes — very short responses often indicate block pages."],"tags":["scraping","nextjs","parsing","comic-walker","anti-bot"],"backgroundTag":null,"analyzedSha":"bed535e0879dc71c5aff6f1e7bd1ac21ede40115","analyzedAt":"2026-08-12T19:29:35.364Z","schemaVersion":2},"datasetVersion":"2026-08-12T23:17:12.415Z"}