{"record":{"id":"50451c96f0ebcee5","repo":"hankcs/HanLP","slug":"encoding-encoding-was-not-supported-tested-suppo","errorCode":null,"errorMessage":"{encoding} encoding was not supported/tested.Supported encodings are '{supported}'","messagePattern":"(.+?) encoding was not supported/tested\\.Supported encodings are '(.+?)'","errorType":"exception","errorClass":"NotImplementedError","httpStatus":null,"severity":"error","filePath":"hanlp/utils/file_read_backwards/file_read_backwards.py","lineNumber":41,"sourceCode":"\n    Args:\n\n    Returns:\n\n    \"\"\"\n\n    def __init__(self, path, encoding=\"utf-8\", chunk_size=io.DEFAULT_BUFFER_SIZE):\n        \"\"\"Constructor for FileReadBackwards.\n\n        Args:\n            path: Path to the file to be read\n            encoding (str): Encoding\n            chunk_size (int): How many bytes to read at a time\n        \"\"\"\n        if encoding.lower() not in supported_encodings:\n            error_message = \"{0} encoding was not supported/tested.\".format(encoding)\n            error_message += \"Supported encodings are '{0}'\".format(\",\".join(supported_encodings))\n            raise NotImplementedError(error_message)\n\n        self.path = path\n        self.encoding = encoding.lower()\n        self.chunk_size = chunk_size\n        self.iterator = FileReadBackwardsIterator(io.open(self.path, mode=\"rb\"), self.encoding, self.chunk_size)\n\n    def __iter__(self):\n        \"\"\"Return its iterator.\"\"\"\n        return self.iterator\n\n    def __enter__(self):\n        return self\n\n    def __exit__(self, exc_type, exc_val, exc_tb):\n        \"\"\"Closes all opened its file handler and propagates all exceptions on exit.\"\"\"\n        self.close()\n        return False\n","sourceCodeStart":23,"sourceCodeEnd":59,"githubUrl":"https://github.com/hankcs/HanLP/blob/ddb1299bddff079e447af52ec12549c50636bfa8/hanlp/utils/file_read_backwards/file_read_backwards.py#L23-L59","documentation":"FileReadBackwards only implements backward reading for encodings where byte-level decoding from a chunk boundary is safe (utf-8, ascii, latin-1, gbk...). Passing an encoding outside supported_encodings raises NotImplementedError listing the supported set.","triggerScenarios":"Constructing FileReadBackwards(path, encoding='utf-16') (or any unsupported codec) — variable/2-byte-prefixed encodings cannot be decoded reliably from arbitrary chunk offsets.","commonSituations":"Reading Windows-generated UTF-16 log files backwards; piping exotic locales (e.g. shift_jis variants) into a tail-like reader.","solutions":["Convert the file to utf-8 first (iconv -f utf-16 -t utf-8), then use FileReadBackwards with encoding='utf-8'.","For ASCII-safe content, read as latin-1 (byte-transparent) and decode lines afterwards.","Patch the library by appending your encoding to supported_encodings only if it is self-synchronizing like utf-8."],"exampleFix":"# before\nFileReadBackwards(p, encoding='utf-16')\n# after\nimport codecs\ntext = open(p, encoding='utf-16').read()\nopen(p + '.u8', 'w', encoding='utf-8').write(text)\nFileReadBackwards(p + '.u8', encoding='utf-8')","handlingStrategy":"validation","validationCode":"from hanlp.utils.file_read_backwards.file_read_backwards import supported_encodings\nassert enc.lower() in supported_encodings, f'convert file to utf-8; supported: {supported_encodings}'","typeGuard":"def encoding_supported(enc):\n    from hanlp.utils.file_read_backwards.file_read_backwards import supported_encodings\n    return enc.lower() in supported_encodings","tryCatchPattern":null,"preventionTips":["Normalize all files to utf-8 at ingest.","Never assume exotic codecs work with chunk-based backward readers."],"tags":["encoding","file-io","not-implemented"],"backgroundTag":"unsupported-encoding","analyzedSha":"ddb1299bddff079e447af52ec12549c50636bfa8","analyzedAt":"2026-08-27T03:36:54.287Z","schemaVersion":2},"datasetVersion":"2026-08-27T08:17:20.692Z"}