MemPalace/mempalace · error · ValueError

{config_path} is not valid UTF-8 — re-save it as UTF-8

Error message

{config_path} is not valid UTF-8 — re-save it as UTF-8

What it means

Raised by DialectConfig.load_config when the entity-mapping JSON file cannot be decoded as UTF-8. The file is opened with encoding='utf-8', so any file saved in a legacy codepage (e.g. Windows-1252, Latin-1) or containing binary bytes raises UnicodeDecodeError, which is re-wrapped as this ValueError with a remediation hint. It fires before any JSON parsing, so the file content is never partially loaded.

Source

Thrown at mempalace/dialect.py:364

        self.lang = lang or current_lang()
        self.aaak_instruction = t("aaak.instruction")
        self.lang_regex = get_regex()

    @classmethod
    def from_config(cls, config_path: str) -> "Dialect":
        """Load entity mappings from a JSON config file.

        Config format:
        {
            "entities": {"Alice": "ALC", "Bob": "BOB"},
            "skip_names": ["Gandalf", "Sherlock"]
        }
        """
        try:
            with open(config_path, "r", encoding="utf-8") as f:
                config = json.load(f)
        except UnicodeDecodeError as exc:
            raise ValueError(f"{config_path} is not valid UTF-8 — re-save it as UTF-8") from exc
        return cls(
            entities=config.get("entities", {}),
            skip_names=config.get("skip_names", []),
            lang=config.get("lang", "en"),
        )

    def save_config(self, config_path: str):
        """Save current entity mappings to a JSON config file."""
        canonical = {}
        seen_codes = set()
        for name, code in self.entity_codes.items():
            if code not in seen_codes and not name.islower():
                canonical[name] = code
                seen_codes.add(code)
            elif code not in seen_codes:
                canonical[name] = code
                seen_codes.add(code)

View on GitHub (pinned to 06cb6987f0)

Solutions

1. Re-save the config file as UTF-8 without BOM issues (VS Code: 'Save with Encoding' > UTF-8; PowerShell: Out-File -Encoding utf8 or Set-Content -Encoding UTF8)
2. If the file is UTF-16, convert it: iconv -f UTF-16 -t UTF-8 config.json > config.utf8.json
3. Validate the file before loading: python -c "open('config.json', encoding='utf-8').read()"
4. If entity names contain non-ASCII characters, ensure the tool that generates the config writes with json.dump(..., ensure_ascii=False, encoding handled by utf-8 file object)

Example fix

# before (PowerShell writes UTF-16)
Get-Content mappings.json | Out-File config.json
# after
Get-Content mappings.json | Out-File config.json -Encoding utf8

# Python-side guard
try:
    cfg = DialectConfig.load_config(path)
except ValueError as e:
    if 'not valid UTF-8' in str(e):
        data = open(path, 'rb').read().decode('utf-16')  # known legacy encoding
        open(path, 'w', encoding='utf-8').write(data)

from pathlib import Path

def assert_utf8(path: str) -> None:
    Path(path).read_bytes().decode("utf-8")  # raises UnicodeDecodeError early with byte offset

try:
    cfg = DialectConfig.load_config(path)
except ValueError as e:
    if "not valid UTF-8" in str(e):
        # re-save/convert the file, then retry once
        ...
    raise

When it happens

Trigger: Calling DialectConfig.load_config(path) (or a CLI/MCP path that loads entity mappings) where the file was saved by an editor defaulting to the system codepage, was created via PowerShell redirection (which writes UTF-16), or was concatenated with binary content.

Common situations: Windows Notepad/Excel exporting 'entities' config as UTF-16 or ANSI; PowerShell Out-File without -Encoding utf8; files copied from old machines with Latin-1 accented names; BOM-prefixed UTF-16 files.

Related errors

• update requires at least one of documents, metadatas, embedd
• query requires exactly one of query_texts or query_embedding
• ChromaBackend has been closed
• pgvector backend requires a local palace path to anchor mism
• milvus_consistency_level must be one of: {allowed}

AI-assisted analysis of MemPalace/mempalace@06cb6987f0 (2026-08-15). Data as JSON: /api/errors/4780951bba2d400d. Report an issue: GitHub.