{"record":{"id":"b875ccac95868b4c","repo":"DIYgod/RSSHub","slug":"techflow-api-still-returned-an-anti-crawler-challe","errorCode":null,"errorMessage":"TechFlow API still returned an anti-crawler challenge after retry.","messagePattern":"TechFlow API still returned an anti-crawler challenge after retry\\.","errorType":"exception","errorClass":"TypeError","httpStatus":503,"severity":"error","filePath":"lib/routes/techflowpost/utils.ts","lineNumber":97,"sourceCode":"        got.get(`${apiRootUrl}${endpoint}`, {\n            searchParams,\n            headers: getHeaders(referer),\n        });\n\n    const { data } = await request();\n    if (typeof data !== 'string') {\n        return data as T;\n    }\n\n    const cookie = getAcwScV2Cookie(data);\n    if (!cookie) {\n        throw new Error('TechFlow API returned an unexpected non-JSON response.');\n    }\n\n    acwScV2Cookie = cookie;\n    const retryResponse = await request();\n    if (typeof retryResponse.data === 'string') {\n        throw new TypeError('TechFlow API still returned an anti-crawler challenge after retry.');\n    }\n\n    return retryResponse.data as T;\n}\n\nfunction getPictureUrl(picture?: string) {\n    if (!picture) {\n        return;\n    }\n    return new URL(picture, uploadRootUrl).href;\n}\n\nfunction getCategories(article: Article) {\n    return [...new Set([article.category?.name, ...(article.labels?.map((label) => label.label) ?? [])].filter(Boolean))] as string[];\n}\n\nfunction getArticleItem(article: Article, content?: string): DataItem {\n    const link = `${rootUrl}/${locale}/article/${article.id}`;","sourceCodeStart":79,"sourceCodeEnd":115,"githubUrl":"https://github.com/DIYgod/RSSHub/blob/bed535e0879dc71c5aff6f1e7bd1ac21ede40115/lib/routes/techflowpost/utils.ts#L79-L115","documentation":"Thrown by requestApi after it solved the TechFlow 'acw_sc__v2' anti-crawler JS challenge, set the derived cookie, and re-issued the request, but the second response body is still a string (i.e. the challenge HTML page, not JSON). It means the cookie-based bypass failed to convince the upstream WAF that the client is a real browser.","triggerScenarios":"The upstream site (TechFlow) returns an HTML page embedding 'var arg1=...' the first time; getAcwScV2Cookie derives acw_sc__v2 and the route retries. If the server still serves the obfuscated challenge HTML on retry (IP flagged, challenge rotated, UA rejected, or arg1 logic outdated), retryResponse.data stays a string and this TypeError fires.","commonSituations":"Running RSSHub from a datacenter/cloud IP that the WAF has greylisted; the acw_sc__v2 derivation algorithm being patched server-side so the generated cookie is rejected; missing/changed browser-like headers (Referer/User-Agent) that the WAF cross-checks; rate-limiting after many requests.","solutions":["Retry the feed request after a longer delay (minutes) from a residential/clean IP, since the failure is usually IP-reputation driven.","Verify getHeaders() still sends a realistic browser User-Agent and the correct Referer (the per-endpoint link) that the WAF expects.","Check whether getAcwScV2ByArg1 still matches the current obfuscation: open the challenge HTML in a real browser, extract arg1, and compare the computed acw_sc__v2 against what the browser sets.","If persistent, route RSSHub egress through a proxy the WAF tolerates, or cache a working acw_sc__v2 cookie from a browser session and inject it into the initial request headers."],"exampleFix":"// before\nacwScV2Cookie = cookie;\nconst retryResponse = await request();\nif (typeof retryResponse.data === 'string') {\n    throw new TypeError('TechFlow API still returned an anti-crawler challenge after retry.');\n}\n\n// after: one bounded retry with a delay, then a clearer error\nacwScV2Cookie = cookie;\nlet retryResponse;\nfor (let attempt = 0; attempt < 2; attempt++) {\n    await new Promise((r) => setTimeout(r, 1000 * (attempt + 1)));\n    retryResponse = await request();\n    if (typeof retryResponse.data !== 'string') {\n        return retryResponse.data as T;\n    }\n}\nthrow new TypeError('TechFlow anti-crawler challenge persists; IP likely flagged or acw_sc__v2 logic outdated.');","handlingStrategy":"retry","validationCode":"// No caller-side data check prevents a WAF challenge, but you can pre-flight reachability\nasync function looksLikeJsonApi(url: string): Promise<boolean> {\n    try {\n        const r = await fetch(url, { headers: { Accept: 'application/json' } });\n        const ct = r.headers.get('content-type') ?? '';\n        return ct.includes('application/json');\n    } catch {\n        return false;\n    }\n}\n// if false, expect a challenge; back off before hitting requestApi","typeGuard":"// detect an anti-crawler challenge body so callers can retry instead of throwing\nfunction isAntiCrawlerChallenge(data: unknown): data is string {\n    return typeof data === 'string' && /var arg1=/.test(data);\n}","tryCatchPattern":"try {\n    return await requestApi<Article>(endpoint, referer, params);\n} catch (e) {\n    if (e instanceof TypeError && /anti-crawler challenge after retry/.test(e.message)) {\n        await sleep(60_000); // WAF reputation usually recovers within minutes\n        return await requestApi<Article>(endpoint, referer, params);\n    }\n    throw e;\n}","preventionTips":["Egress RSSHub from a residential/clean IP rather than a flagged datacenter range.","Keep getHeaders() realistic: full browser User-Agent and a matching Referer for every endpoint.","Periodically verify getAcwScV2ByArg1 still reproduces the browser's acw_sc__v2 for a sample arg1.","Rate-limit your own polling so the WAF does not greylist the source IP."],"tags":["anti-crawler","network","retry","waf"],"backgroundTag":null,"analyzedSha":"bed535e0879dc71c5aff6f1e7bd1ac21ede40115","analyzedAt":"2026-08-12T19:29:35.364Z","schemaVersion":2},"datasetVersion":"2026-08-12T23:17:12.415Z"}