{"record":{"id":"7dfb3e66621c5e9b","repo":"apify/crawlee","slug":"resource-request-url-served-with-unsupported-ch","errorCode":null,"errorMessage":"Resource ${request.url} served with unsupported charset/encoding: ${encoding}","messagePattern":"Resource (.+?) served with unsupported charset/encoding: (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"packages/http-crawler/src/internals/http-crawler.ts","lineNumber":835,"sourceCode":"                .decodeStream(encoding)\n                .on('error', (err: Error) => encodeStream.emit('error', err));\n            const reencodedBody = response.body\n                ? Readable.toWeb(\n                      Readable.from(\n                          Readable.fromWeb(response.body as any)\n                              .pipe(decodeStream)\n                              .pipe(encodeStream),\n                      ),\n                  )\n                : null;\n\n            return {\n                response: new ResponseWithUrl(reencodedBody as any, response),\n                encoding: utf8,\n            };\n        }\n\n        throw new Error(`Resource ${request.url} served with unsupported charset/encoding: ${encoding}`);\n    }\n\n    /**\n     * Checks and extends supported mime types\n     */\n    private extendSupportedMimeTypes(additionalMimeTypes: (string | RequestLike | ResponseLike)[]) {\n        for (const mimeType of additionalMimeTypes) {\n            if (mimeType === '*/*') {\n                this.#supportedMimeTypes.add(mimeType);\n                continue;\n            }\n\n            try {\n                const parsedType = contentTypeParser.parse(mimeType);\n                this.#supportedMimeTypes.add(parsedType.type);\n            } catch (err) {\n                throw new Error(`Can not parse mime type ${mimeType} from \"options.additionalMimeTypes\".`);\n            }","sourceCodeStart":817,"sourceCodeEnd":853,"githubUrl":"https://github.com/apify/crawlee/blob/dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c/packages/http-crawler/src/internals/http-crawler.ts#L817-L853","documentation":"encodeResponse decodes the response body into UTF-8 using supported encodings; if the resource's charset/encoding is neither among the supported set nor can it be re-encoded, it throws this error naming the URL and offending encoding. The library will not process bodies it cannot reliably decode.","triggerScenarios":"Response Content-Type or meta tags declare an exotic charset (e.g. x-mac-cyrillic, iso-2022-jp variants, or a garbage/unknown value) while the crawler processes the body via encodeResponse.","commonSituations":"Legacy sites with unusual encodings; misconfigured servers sending a wrong charset parameter; non-Latin legacy systems (old Japanese/Chinese/DOS encodings); pages with invalid meta charset attributes.","solutions":["If the actual bytes are decodable, set `forceResponseEncoding` (e.g. 'utf-8' or the correct encoding) to override the declared charset.","Pre-process with a proxy/worker that normalizes the encoding, or fetch and decode manually with iconv-lite in a custom requestFunction.","Check whether `suggestResponseEncoding` is set to an unsupported value and correct it.","Skip the resource if its content is not needed (filter by content type/URL pattern before processing)."],"exampleFix":"// before\nnew HttpCrawler({}); // server sends charset=x-mac-cyrillic\n\n// after\nnew HttpCrawler({ forceResponseEncoding: 'windows-1251' });","handlingStrategy":"fallback","validationCode":"const res = await got(url, { throwHttpErrors: false });\nconst ct = res.headers['content-type'] ?? '';\nconst m = /charset=([^;]+)/i.exec(ct);\nif (m && !iconv.encodingExists(m[1].trim())) {\n  log.warning(`Unsupported charset ${m[1]}; will override`);\n}","typeGuard":"function hasSupportedCharset(charset: string | undefined): charset is string {\n  return !!charset && iconv.encodingExists(charset);\n}","tryCatchPattern":"try {\n  await crawler.run(requests);\n} catch (err) {\n  if (err instanceof Error && err.message.includes('unsupported charset/encoding')) {\n    log.warning(`Skipping undecodable resource: ${err.message}`);\n    return;\n  }\n  throw err;\n}","preventionTips":["Set forceResponseEncoding for sites known to declare exotic or wrong charsets.","Detect target-site charsets during development (curl -I) and configure accordingly.","Normalize legacy encodings upstream if crawling archives of old sites.","Log the URL + encoding from the message to build a skip-list."],"tags":["encoding","charset","http","internationalization"],"backgroundTag":"unsupported-charset","analyzedSha":"dbe57fb09ca607ad59dcf998f3925ef9ac3bb26c","analyzedAt":"2026-08-30T22:22:28.328Z","schemaVersion":2},"datasetVersion":"2026-08-30T23:17:21.991Z"}