{"record":{"id":"3dafc80ef02a526c","repo":"MemPalace/mempalace","slug":"content-contains-null-bytes","errorCode":null,"errorMessage":"content contains null bytes","messagePattern":"content contains null bytes","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"mempalace/config.py","lineNumber":217,"sourceCode":"def sanitize_iso_date(value, field_name: str = \"date\"):\n    \"\"\"Backward-compatible wrapper for ISO temporal validation.\n\n    Historically this accepted only full dates. It now also accepts canonical\n    UTC datetimes, but the old name is kept so existing imports continue to\n    work.\n    \"\"\"\n\n    return sanitize_iso_temporal(value, field_name)\n\n\ndef sanitize_content(value: str, max_length: int = 100_000) -> str:\n    \"\"\"Validate drawer/diary content length.\"\"\"\n    if not isinstance(value, str) or not value.strip():\n        raise ValueError(\"content must be a non-empty string\")\n    if len(value) > max_length:\n        raise ValueError(f\"content exceeds maximum length of {max_length} characters\")\n    if \"\\x00\" in value:\n        raise ValueError(\"content contains null bytes\")\n    return strip_lone_surrogates(value)\n\n\nDEFAULT_PALACE_PATH = os.path.expanduser(\"~/.mempalace/palace\")\nDEFAULT_COLLECTION_NAME = \"mempalace_drawers\"\nDEFAULT_BACKEND = \"chroma\"\nDEFAULT_MILVUS_CONSISTENCY_LEVEL = \"Strong\"\n_MILVUS_CONSISTENCY_LEVELS = {\n    \"strong\": \"Strong\",\n    \"session\": \"Session\",\n    \"bounded\": \"Bounded\",\n    \"eventually\": \"Eventually\",\n}\n\n# How many timestamped palace backups to retain before the oldest are\n# pruned. Applies to the accumulating backups written by ``mempalace\n# migrate`` and ``mempalace repair max-seq-id`` — see\n# ``MempalaceConfig.max_backups``.","sourceCodeStart":199,"sourceCodeEnd":235,"githubUrl":"https://github.com/MemPalace/mempalace/blob/06cb6987f02610784fefbad4b2bd5d026d164ba6/mempalace/config.py#L199-L235","documentation":"Raised by sanitize_content() when drawer/diary content contains a NUL byte (\\x00). NUL bytes break the storage layer (SQLite TEXT, ChromaDB serialization) and would corrupt the verbatim record, so any content containing one is rejected before storage. Encountering this usually means binary data or a decoding error reached the content pipeline.","triggerScenarios":"Passing content decoded leniently from binary (errors='ignore' can still keep \\x00), or content read from a corrupted/binary file — e.g. ingest of a mis-detected .bin transcript.","commonSituations":"Miner walking a directory that contains binary files; UTF-16 files decoded as UTF-8; corrupted downloads; text fields in transcripts that embedded raw bytes.","solutions":["Strip NUL bytes before saving if lossy cleanup is acceptable: content.replace(\"\\x00\", \"\")","Better: reject binary files at ingestion (check for \\x00 in the first chunk, like the classic binary-sniff heuristic) and decode text files strictly","Trace which file produced the content and fix its decode mode (e.g. encoding='utf-16' for UTF-16 sources)"],"exampleFix":"# before\ncontent = path.read_bytes().decode(\"utf-8\", errors=\"ignore\")\nsave_drawer(wing, room, content)\n\n# after\nraw = path.read_bytes()\nif b\"\\x00\" in raw[:8192]:\n    raise ValueError(f\"{path} is binary; skipping\")\nsave_drawer(wing, room, raw.decode(\"utf-8\"))","handlingStrategy":"validation","validationCode":"# Binary sniff before ingest (classic heuristic):\nraw = path.read_bytes()\nif b\"\\x00\" in raw[:8192]:\n    raise SystemExit(f\"{path} looks binary; skip or fix decoding\")\ncontent = raw.decode(\"utf-8\")  # strict","typeGuard":"def is_nul_free_text(content: str) -> bool:\n    return isinstance(content, str) and \"\\x00\" not in content","tryCatchPattern":"try:\n    safe = sanitize_content(content)\nexcept ValueError as exc:\n    if \"null bytes\" in str(exc):\n        log.warning(\"stripped NUL bytes from content\")\n        safe = sanitize_content(content.replace(\"\\x00\", \"\"))\n    else:\n        raise","preventionTips":["Sniff for NUL bytes in the first 8KB to reject binary files at the miner boundary","Decode with errors='strict' so corruption surfaces early","Detect the real encoding (UTF-16 etc.) instead of lossy passthrough"],"tags":["validation","content","binary-data","mempalace"],"backgroundTag":null,"analyzedSha":"06cb6987f02610784fefbad4b2bd5d026d164ba6","analyzedAt":"2026-08-15T03:03:36.213Z","schemaVersion":2},"datasetVersion":"2026-08-15T22:17:37.221Z"}