agentscope-ai/agentscope · error · ValueError

Failed to decode {filename!r} as {self.encoding!r}: {e}

Error message

Failed to decode {filename!r} as {self.encoding!r}: {e}

What it means

The text parser tried to decode file bytes using the configured encoding (default utf-8) and hit a UnicodeDecodeError. This variant occurs when the input is an existing file path on disk.

Source

Thrown at src/agentscope/rag/_parser/_text.py:104

                :attr:`Section.source`.

        Returns:
            `list[Section]`:
                Always a one-element list containing the entire file
                contents.

        Raises:
            `ValueError`: If the bytes cannot be decoded with the
                configured encoding.
        """
        if isinstance(file, str):
            if os.path.isfile(file):
                with open(file, "rb") as fp:
                    raw = fp.read()
                try:
                    text = raw.decode(self.encoding)
                except UnicodeDecodeError as e:
                    raise ValueError(
                        f"Failed to decode {filename!r} as "
                        f"{self.encoding!r}: {e}",
                    ) from e
            else:
                text = file
        else:
            try:
                text = file.decode(self.encoding)
            except UnicodeDecodeError as e:
                raise ValueError(
                    f"Failed to decode {filename!r} as "
                    f"{self.encoding!r}: {e}",
                ) from e

        return [
            Section(
                content=TextBlock(text=text),
                source=filename,

View on GitHub (pinned to e90f1c7592)

Solutions

1. Detect and pass the correct encoding: TextParser(encoding=charset_normalizer.detect(raw)['encoding'])
2. Convert the file to UTF-8 once: iconv -f cp1252 -t utf-8
3. Use errors-tolerant preprocessing or 'latin-1' which never fails
4. Skip non-text files when walking directories

Example fix

# before
parser = TextParser()  # utf-8
parser.parse('notes.txt')
# after
parser = TextParser(encoding='cp1252')
parser.parse('notes.txt')

from charset_normalizer import from_path
match = from_path('file.txt').best()
enc = match.encoding if match else 'utf-8'

try:
    parser = TextParser(); parser.parse(path)
except ValueError:
    parser = TextParser(encoding='latin-1'); parser.parse(path)

When it happens

Trigger: TextParser.parse('/path/file.txt') or build_index over a file containing bytes invalid in self.encoding, e.g. Latin-1/CP1252 files with smart quotes parsed as utf-8.

Common situations: Windows-authored text files (CP1252), mixed-encoding corpora, or binary files with a .txt extension fed into a RAG indexing job.

Understand the failure class

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• module {__name__!r} has no attribute {name!r}
• AGENTSCOPE_WORKER_BOOTSTRAP must be in 'module:attr' form, g
• Knowledge base {knowledge_base_id!r} not found.
• Credential {record.data.embedding_model_config.credential_id
• DimensionPolicy: kind=ANY requires dimension=None, got dimen

AI-assisted analysis of agentscope-ai/agentscope@e90f1c7592 (2026-08-28). Data as JSON: /api/errors/4a842433290ead14. Report an issue: GitHub.