apify/crawlee · error · Error

Resource ${request.url} served with unsupported charset/enco

Error message

Resource ${request.url} served with unsupported charset/encoding: ${encoding}

What it means

encodeResponse decodes the response body into UTF-8 using supported encodings; if the resource's charset/encoding is neither among the supported set nor can it be re-encoded, it throws this error naming the URL and offending encoding. The library will not process bodies it cannot reliably decode.

Source

Thrown at packages/http-crawler/src/internals/http-crawler.ts:835

                .decodeStream(encoding)
                .on('error', (err: Error) => encodeStream.emit('error', err));
            const reencodedBody = response.body
                ? Readable.toWeb(
                      Readable.from(
                          Readable.fromWeb(response.body as any)
                              .pipe(decodeStream)
                              .pipe(encodeStream),
                      ),
                  )
                : null;

            return {
                response: new ResponseWithUrl(reencodedBody as any, response),
                encoding: utf8,
            };
        }

        throw new Error(`Resource ${request.url} served with unsupported charset/encoding: ${encoding}`);
    }

    /**
     * Checks and extends supported mime types
     */
    private extendSupportedMimeTypes(additionalMimeTypes: (string | RequestLike | ResponseLike)[]) {
        for (const mimeType of additionalMimeTypes) {
            if (mimeType === '*/*') {
                this.#supportedMimeTypes.add(mimeType);
                continue;
            }

            try {
                const parsedType = contentTypeParser.parse(mimeType);
                this.#supportedMimeTypes.add(parsedType.type);
            } catch (err) {
                throw new Error(`Can not parse mime type ${mimeType} from "options.additionalMimeTypes".`);
            }

View on GitHub (pinned to dbe57fb09c)

Solutions

1. If the actual bytes are decodable, set `forceResponseEncoding` (e.g. 'utf-8' or the correct encoding) to override the declared charset.
2. Pre-process with a proxy/worker that normalizes the encoding, or fetch and decode manually with iconv-lite in a custom requestFunction.
3. Check whether `suggestResponseEncoding` is set to an unsupported value and correct it.
4. Skip the resource if its content is not needed (filter by content type/URL pattern before processing).

Example fix

// before
new HttpCrawler({}); // server sends charset=x-mac-cyrillic

// after
new HttpCrawler({ forceResponseEncoding: 'windows-1251' });

const res = await got(url, { throwHttpErrors: false });
const ct = res.headers['content-type'] ?? '';
const m = /charset=([^;]+)/i.exec(ct);
if (m && !iconv.encodingExists(m[1].trim())) {
  log.warning(`Unsupported charset ${m[1]}; will override`);
}

function hasSupportedCharset(charset: string | undefined): charset is string {
  return !!charset && iconv.encodingExists(charset);
}

try {
  await crawler.run(requests);
} catch (err) {
  if (err instanceof Error && err.message.includes('unsupported charset/encoding')) {
    log.warning(`Skipping undecodable resource: ${err.message}`);
    return;
  }
  throw err;
}

When it happens

Trigger: Response Content-Type or meta tags declare an exotic charset (e.g. x-mac-cyrillic, iso-2022-jp variants, or a garbage/unknown value) while the crawler processes the body via encodeResponse.

Common situations: Legacy sites with unusual encodings; misconfigured servers sending a wrong charset parameter; non-Latin legacy systems (old Japanese/Chinese/DOS encodings); pages with invalid meta charset attributes.

Related errors

• Could not parse cookie header string: ${cookieHeaderString}
• Request with GET method cannot have a payload.
• Cannot fetch a request list from ${requestsFromUrl}: ${err}
• Cannot fetch a request list from ${requestsFromUrl}: ${err}
• The HTTP method CONNECT is not supported by the GotScrapingH

AI-assisted analysis of apify/crawlee@dbe57fb09c (2026-08-30). Data as JSON: /api/errors/7dfb3e66621c5e9b. Report an issue: GitHub.