MemPalace/mempalace · error · ValueError

content contains null bytes

Error message

content contains null bytes

What it means

put_artifact rejects content containing NUL characters ('\\x00'). Artifacts are stored as exact UTF-8 text and verified by sha256, and embedded NULs corrupt text columns and digest round-trips, so the write fails loudly rather than truncating. This check runs before surrogate stripping and size measurement.

Source

Thrown at mempalace/logstream.py:528

        created_by: str,
        metadata: dict = None,
    ) -> dict:
        """Store exact artifact content (v1: UTF-8 text only).

        Returns the artifact record without echoing ``content`` back —
        callers already hold the content; readers use :meth:`get_artifact`.
        For ``kind=patch``, a ``warnings`` list is included when the diff
        looks unappliable (missing trailing newline, CRLF endings); the
        content itself is still stored verbatim.
        """
        if not isinstance(kind, str) or kind not in ARTIFACT_KINDS:
            allowed = ", ".join(sorted(ARTIFACT_KINDS))
            raise ValueError(f"kind={kind!r} is not one of: {allowed}")
        created_by = _sanitize_routing(created_by, "created_by")
        if not isinstance(content, str) or not content:
            raise ValueError("content must be a non-empty string")
        if "\x00" in content:
            raise ValueError("content contains null bytes")
        content = strip_lone_surrogates(content)
        raw = content.encode("utf-8")
        if len(raw) > self.max_artifact_bytes:
            raise ValueError(
                f"content is {len(raw)} bytes; maximum is {self.max_artifact_bytes} bytes"
            )
        metadata_json = _sanitize_metadata(metadata)

        artifact_id = _new_id("art")
        created_at = _utc_now_iso()
        digest = sha256(raw).hexdigest()

        with self._lock:
            conn = self._conn()
            with conn:
                conn.execute(
                    "INSERT INTO artifacts (id, kind, sha256, size_bytes, content,"
                    " created_by, created_at, metadata_json, origin_replica)"

View on GitHub (pinned to 06cb6987f0)

Solutions

1. Detect binary first and reject/handle: if b'\\x00' in raw: treat as binary (v1 has no binary artifact kind).
2. Strip NULs if they are padding artifacts: content=content.replace('\\x00', '').
3. Base64-encode truly binary payloads into a 'note'/'json' artifact and record the encoding in metadata.

Example fix

// before
art = ls.put_artifact(kind="file", content=raw.decode("utf-8", errors="replace"), ...)
// after
import base64
art = ls.put_artifact(kind="json", content=json.dumps({"encoding": "base64", "data": base64.b64encode(raw).decode()}), ...)

def is_text_payload(raw: bytes) -> bool:
    return b"\x00" not in raw

if not is_text_payload(raw):
    raise TypeError("binary payload; encode as base64 in a json/note artifact")
content = raw.decode("utf-8")

def is_nul_free_text(content) -> bool:
    return isinstance(content, str) and "\x00" not in content

try:
    art = ls.put_artifact(kind=kind, content=content, ...)
except ValueError as e:
    if "null bytes" in str(e):
        art = ls.put_artifact(kind="json", content=json.dumps({"encoding": "base64", "data": __import__("base64").b64encode(content.encode()).decode()}), ...)
    else:
        raise

When it happens

Trigger: put_artifact(kind='file', content=data.decode('utf-8')) where data has embedded NULs (binary misread as text); fixed-width buffers with NUL padding; protocol captures containing length-prefixed frames with NUL separators.

Common situations: Accidentally classifying binary as a text artifact (e.g. .db, .png mislabeled kind='file'); reading C structs via struct.unpack and joining raw bytes; Windows UTF-16 text decoded permissively leaving NULs.

Related errors

• {field_name} contains null bytes
• kind={kind!r} is not one of: {allowed}
• content must be a non-empty string
• content is {len(raw)} bytes; maximum is {self.max_artifact_b
• {field_name} must be a non-empty string

AI-assisted analysis of MemPalace/mempalace@06cb6987f0 (2026-08-15). Data as JSON: /api/errors/8c2573208c51f880. Report an issue: GitHub.