{"record":{"id":"4780951bba2d400d","repo":"MemPalace/mempalace","slug":"config-path-is-not-valid-utf-8-re-save-it-as-u","errorCode":null,"errorMessage":"{config_path} is not valid UTF-8 — re-save it as UTF-8","messagePattern":"(.+?) is not valid UTF-8 — re-save it as UTF-8","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"mempalace/dialect.py","lineNumber":364,"sourceCode":"        self.lang = lang or current_lang()\n        self.aaak_instruction = t(\"aaak.instruction\")\n        self.lang_regex = get_regex()\n\n    @classmethod\n    def from_config(cls, config_path: str) -> \"Dialect\":\n        \"\"\"Load entity mappings from a JSON config file.\n\n        Config format:\n        {\n            \"entities\": {\"Alice\": \"ALC\", \"Bob\": \"BOB\"},\n            \"skip_names\": [\"Gandalf\", \"Sherlock\"]\n        }\n        \"\"\"\n        try:\n            with open(config_path, \"r\", encoding=\"utf-8\") as f:\n                config = json.load(f)\n        except UnicodeDecodeError as exc:\n            raise ValueError(f\"{config_path} is not valid UTF-8 — re-save it as UTF-8\") from exc\n        return cls(\n            entities=config.get(\"entities\", {}),\n            skip_names=config.get(\"skip_names\", []),\n            lang=config.get(\"lang\", \"en\"),\n        )\n\n    def save_config(self, config_path: str):\n        \"\"\"Save current entity mappings to a JSON config file.\"\"\"\n        canonical = {}\n        seen_codes = set()\n        for name, code in self.entity_codes.items():\n            if code not in seen_codes and not name.islower():\n                canonical[name] = code\n                seen_codes.add(code)\n            elif code not in seen_codes:\n                canonical[name] = code\n                seen_codes.add(code)\n","sourceCodeStart":346,"sourceCodeEnd":382,"githubUrl":"https://github.com/MemPalace/mempalace/blob/06cb6987f02610784fefbad4b2bd5d026d164ba6/mempalace/dialect.py#L346-L382","documentation":"Raised by DialectConfig.load_config when the entity-mapping JSON file cannot be decoded as UTF-8. The file is opened with encoding='utf-8', so any file saved in a legacy codepage (e.g. Windows-1252, Latin-1) or containing binary bytes raises UnicodeDecodeError, which is re-wrapped as this ValueError with a remediation hint. It fires before any JSON parsing, so the file content is never partially loaded.","triggerScenarios":"Calling DialectConfig.load_config(path) (or a CLI/MCP path that loads entity mappings) where the file was saved by an editor defaulting to the system codepage, was created via PowerShell redirection (which writes UTF-16), or was concatenated with binary content.","commonSituations":"Windows Notepad/Excel exporting 'entities' config as UTF-16 or ANSI; PowerShell Out-File without -Encoding utf8; files copied from old machines with Latin-1 accented names; BOM-prefixed UTF-16 files.","solutions":["Re-save the config file as UTF-8 without BOM issues (VS Code: 'Save with Encoding' > UTF-8; PowerShell: Out-File -Encoding utf8 or Set-Content -Encoding UTF8)","If the file is UTF-16, convert it: iconv -f UTF-16 -t UTF-8 config.json > config.utf8.json","Validate the file before loading: python -c \"open('config.json', encoding='utf-8').read()\"","If entity names contain non-ASCII characters, ensure the tool that generates the config writes with json.dump(..., ensure_ascii=False, encoding handled by utf-8 file object)"],"exampleFix":"# before (PowerShell writes UTF-16)\nGet-Content mappings.json | Out-File config.json\n# after\nGet-Content mappings.json | Out-File config.json -Encoding utf8\n\n# Python-side guard\ntry:\n    cfg = DialectConfig.load_config(path)\nexcept ValueError as e:\n    if 'not valid UTF-8' in str(e):\n        data = open(path, 'rb').read().decode('utf-16')  # known legacy encoding\n        open(path, 'w', encoding='utf-8').write(data)","handlingStrategy":"validation","validationCode":"from pathlib import Path\n\ndef assert_utf8(path: str) -> None:\n    Path(path).read_bytes().decode(\"utf-8\")  # raises UnicodeDecodeError early with byte offset","typeGuard":null,"tryCatchPattern":"try:\n    cfg = DialectConfig.load_config(path)\nexcept ValueError as e:\n    if \"not valid UTF-8\" in str(e):\n        # re-save/convert the file, then retry once\n        ...\n    raise","preventionTips":["Always write config files with encoding='utf-8' in any script that generates them","In PowerShell use Out-File -Encoding utf8, never bare Out-File (UTF-16)","Keep entity config under version control so a corrupted save can be reverted","Validate the file decodes as UTF-8 before loading it in setup scripts"],"tags":["encoding","utf-8","config","dialect","file-io"],"backgroundTag":null,"analyzedSha":"06cb6987f02610784fefbad4b2bd5d026d164ba6","analyzedAt":"2026-08-15T03:03:36.213Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}