hankcs/HanLP · error · NotImplementedError

{encoding} encoding was not supported/tested.Supported encod

Error message

{encoding} encoding was not supported/tested.Supported encodings are '{supported}'

What it means

FileReadBackwards only implements backward reading for encodings where byte-level decoding from a chunk boundary is safe (utf-8, ascii, latin-1, gbk...). Passing an encoding outside supported_encodings raises NotImplementedError listing the supported set.

Source

Thrown at hanlp/utils/file_read_backwards/file_read_backwards.py:41

    Args:

    Returns:

    """

    def __init__(self, path, encoding="utf-8", chunk_size=io.DEFAULT_BUFFER_SIZE):
        """Constructor for FileReadBackwards.

        Args:
            path: Path to the file to be read
            encoding (str): Encoding
            chunk_size (int): How many bytes to read at a time
        """
        if encoding.lower() not in supported_encodings:
            error_message = "{0} encoding was not supported/tested.".format(encoding)
            error_message += "Supported encodings are '{0}'".format(",".join(supported_encodings))
            raise NotImplementedError(error_message)

        self.path = path
        self.encoding = encoding.lower()
        self.chunk_size = chunk_size
        self.iterator = FileReadBackwardsIterator(io.open(self.path, mode="rb"), self.encoding, self.chunk_size)

    def __iter__(self):
        """Return its iterator."""
        return self.iterator

    def __enter__(self):
        return self

    def __exit__(self, exc_type, exc_val, exc_tb):
        """Closes all opened its file handler and propagates all exceptions on exit."""
        self.close()
        return False

View on GitHub (pinned to ddb1299bdd)

Solutions

Convert the file to utf-8 first (iconv -f utf-16 -t utf-8), then use FileReadBackwards with encoding='utf-8'.
For ASCII-safe content, read as latin-1 (byte-transparent) and decode lines afterwards.
Patch the library by appending your encoding to supported_encodings only if it is self-synchronizing like utf-8.

Example fix

# before
FileReadBackwards(p, encoding='utf-16')
# after
import codecs
text = open(p, encoding='utf-16').read()
open(p + '.u8', 'w', encoding='utf-8').write(text)
FileReadBackwards(p + '.u8', encoding='utf-8')

Defensive patterns

Strategy: validation

Validate before calling

from hanlp.utils.file_read_backwards.file_read_backwards import supported_encodings
assert enc.lower() in supported_encodings, f'convert file to utf-8; supported: {supported_encodings}'

Type guard

def encoding_supported(enc):
    from hanlp.utils.file_read_backwards.file_read_backwards import supported_encodings
    return enc.lower() in supported_encodings

Prevention

Normalize all files to utf-8 at ingest.
Never assume exotic codecs work with chunk-based backward readers.

When it happens

Trigger: Constructing FileReadBackwards(path, encoding='utf-16') (or any unsupported codec) — variable/2-byte-prefixed encodings cannot be decoded reliably from arbitrary chunk offsets.

Common situations: Reading Windows-generated UTF-16 log files backwards; piping exotic locales (e.g. shift_jis variants) into a tail-like reader.

Related errors

AI-assisted analysis of hankcs/HanLP@ddb1299bdd (2026-08-27). Data as JSON: /api/errors/50451c96f0ebcee5. Report an issue: GitHub.