{"record":{"id":"fa3e9aeacb7678ae","repo":"MemPalace/mempalace","slug":"field-name-contains-null-bytes-fa3e9a","errorCode":null,"errorMessage":"{field_name} contains null bytes","messagePattern":"(.+?) contains null bytes","errorType":"validation","errorClass":"ValueError","httpStatus":null,"severity":"error","filePath":"mempalace/logstream.py","lineNumber":148,"sourceCode":"\n\ndef _sanitize_status(value) -> Optional[str]:\n    if value is None or value == \"\":\n        return None\n    if not isinstance(value, str) or value not in EVENT_STATUSES:\n        allowed = \", \".join(sorted(EVENT_STATUSES))\n        raise ValueError(f\"status={value!r} is not one of: {allowed}\")\n    return value\n\n\ndef _sanitize_body(value, max_bytes: int, field_name: str = \"body\") -> str:\n    \"\"\"Validate verbatim payload text. Empty is allowed; ``None`` becomes ''.\"\"\"\n    if value is None:\n        return \"\"\n    if not isinstance(value, str):\n        raise ValueError(f\"{field_name} must be a string\")\n    if \"\\x00\" in value:\n        raise ValueError(f\"{field_name} contains null bytes\")\n    value = strip_lone_surrogates(value)\n    size = len(value.encode(\"utf-8\"))\n    if size > max_bytes:\n        raise ValueError(f\"{field_name} is {size} bytes; maximum is {max_bytes} bytes\")\n    return value\n\n\ndef _sanitize_metadata(value) -> str:\n    \"\"\"Validate optional metadata dict and return its canonical JSON text.\"\"\"\n    if value is None:\n        return \"{}\"\n    if not isinstance(value, dict):\n        raise ValueError(\"metadata must be an object\")\n    try:\n        encoded = json.dumps(value, ensure_ascii=False, sort_keys=True)\n    except (TypeError, ValueError) as exc:\n        raise ValueError(f\"metadata is not JSON-serializable: {exc}\") from None\n    if len(encoded.encode(\"utf-8\")) > MAX_METADATA_BYTES:","sourceCodeStart":130,"sourceCodeEnd":166,"githubUrl":"https://github.com/MemPalace/mempalace/blob/06cb6987f02610784fefbad4b2bd5d026d164ba6/mempalace/logstream.py#L130-L166","documentation":"The event body (or note/metadata-bearing field routed through _sanitize_body) contains a NUL character '\\x00'. SQLite text and the verbatim-storage contract cannot safely carry embedded NULs, so the value is rejected rather than silently truncated. This fires before the size check.","triggerScenarios":"append_event(body='line1\\x00line2'); body built from binary data that was decoded with errors='replace' but still contains NULs; content copied from a fixed-width/padded buffer with NUL padding past the terminator.","commonSituations":"Reading C-strings or binary protocol output into Python; log lines concatenated from substr() slices in SQLite or from mmap'd files; artifacts produced on Windows with odd encodings.","solutions":["Strip NULs before calling: body = body.replace('\\x00', '').","Fix the upstream reader to stop at the NUL terminator (e.g. slice to buf.index(b'\\x00')).","If the payload really is binary, base64-encode it into a text body instead."],"exampleFix":"// before\nbody = raw.decode(\"utf-8\")  # raw contains b\"a\\x00b\"\nevt = ls.append_event(type=\"log.entry\", body=body, ...)\n// after\nbody = raw.decode(\"utf-8\").replace(\"\\x00\", \"\")\nevt = ls.append_event(type=\"log.entry\", body=body, ...)","handlingStrategy":"validation","validationCode":"def safe_body(text: str) -> str:\n    if \"\\x00\" in text:\n        raise ValueError(\"payload contains NUL bytes; strip or encode before storing\")\n    return text\n\n# or strip unconditionally when NULs are known padding artifacts:\nbody = body.replace(\"\\x00\", \"\")","typeGuard":"def is_nul_free(text) -> bool:\n    return isinstance(text, str) and \"\\x00\" not in text","tryCatchPattern":"try:\n    evt = ls.append_event(..., body=body)\nexcept ValueError as e:\n    if \"null bytes\" in str(e):\n        evt = ls.append_event(..., body=body.replace(\"\\x00\", \"\"))\n    else:\n        raise","preventionTips":["Check b'\\\\x00' at the point of reading binary sources, before decoding to text.","Base64-encode binary payloads instead of forcing them through text bodies.","Add a shared sanitize_text() helper and use it for every body/content field in your producer."],"tags":["validation","logstream","null-bytes","sanitization"],"backgroundTag":null,"analyzedSha":"06cb6987f02610784fefbad4b2bd5d026d164ba6","analyzedAt":"2026-08-15T03:03:36.213Z","schemaVersion":2},"datasetVersion":"2026-08-15T17:31:12.345Z"}