MemPalace/mempalace · error · ValueError

content contains null bytes

Error message

content contains null bytes

What it means

Raised by sanitize_content() when drawer/diary content contains a NUL byte (\x00). NUL bytes break the storage layer (SQLite TEXT, ChromaDB serialization) and would corrupt the verbatim record, so any content containing one is rejected before storage. Encountering this usually means binary data or a decoding error reached the content pipeline.

Source

Thrown at mempalace/config.py:217

def sanitize_iso_date(value, field_name: str = "date"):
    """Backward-compatible wrapper for ISO temporal validation.

    Historically this accepted only full dates. It now also accepts canonical
    UTC datetimes, but the old name is kept so existing imports continue to
    work.
    """

    return sanitize_iso_temporal(value, field_name)


def sanitize_content(value: str, max_length: int = 100_000) -> str:
    """Validate drawer/diary content length."""
    if not isinstance(value, str) or not value.strip():
        raise ValueError("content must be a non-empty string")
    if len(value) > max_length:
        raise ValueError(f"content exceeds maximum length of {max_length} characters")
    if "\x00" in value:
        raise ValueError("content contains null bytes")
    return strip_lone_surrogates(value)


DEFAULT_PALACE_PATH = os.path.expanduser("~/.mempalace/palace")
DEFAULT_COLLECTION_NAME = "mempalace_drawers"
DEFAULT_BACKEND = "chroma"
DEFAULT_MILVUS_CONSISTENCY_LEVEL = "Strong"
_MILVUS_CONSISTENCY_LEVELS = {
    "strong": "Strong",
    "session": "Session",
    "bounded": "Bounded",
    "eventually": "Eventually",
}

# How many timestamped palace backups to retain before the oldest are
# pruned. Applies to the accumulating backups written by ``mempalace
# migrate`` and ``mempalace repair max-seq-id`` — see
# ``MempalaceConfig.max_backups``.

View on GitHub (pinned to 06cb6987f0)

Solutions

1. Strip NUL bytes before saving if lossy cleanup is acceptable: content.replace("\x00", "")
2. Better: reject binary files at ingestion (check for \x00 in the first chunk, like the classic binary-sniff heuristic) and decode text files strictly
3. Trace which file produced the content and fix its decode mode (e.g. encoding='utf-16' for UTF-16 sources)

Example fix

# before
content = path.read_bytes().decode("utf-8", errors="ignore")
save_drawer(wing, room, content)

# after
raw = path.read_bytes()
if b"\x00" in raw[:8192]:
    raise ValueError(f"{path} is binary; skipping")
save_drawer(wing, room, raw.decode("utf-8"))

# Binary sniff before ingest (classic heuristic):
raw = path.read_bytes()
if b"\x00" in raw[:8192]:
    raise SystemExit(f"{path} looks binary; skip or fix decoding")
content = raw.decode("utf-8")  # strict

def is_nul_free_text(content: str) -> bool:
    return isinstance(content, str) and "\x00" not in content

try:
    safe = sanitize_content(content)
except ValueError as exc:
    if "null bytes" in str(exc):
        log.warning("stripped NUL bytes from content")
        safe = sanitize_content(content.replace("\x00", ""))
    else:
        raise

When it happens

Trigger: Passing content decoded leniently from binary (errors='ignore' can still keep \x00), or content read from a corrupted/binary file — e.g. ingest of a mis-detected .bin transcript.

Common situations: Miner walking a directory that contains binary files; UTF-16 files decoded as UTF-8; corrupted downloads; text fields in transcripts that embedded raw bytes.

Related errors

• {field_name} contains null bytes
• content must be a non-empty string
• content exceeds maximum length of {max_length} characters
• pass inline text or a file, not both
• --metadata is not valid JSON: {exc}

AI-assisted analysis of MemPalace/mempalace@06cb6987f0 (2026-08-15). Data as JSON: /api/errors/3dafc80ef02a526c. Report an issue: GitHub.