{"record":{"id":"3510494159714f62","repo":"nautechsystems/nautilus_trader","slug":"c-string-contains-invalid-utf-8","errorCode":null,"errorMessage":"C string contains invalid UTF-8","messagePattern":"C string contains invalid UTF-8","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/core/src/ffi/string.rs","lineNumber":129,"sourceCode":"}\n\n/// Convert a C string pointer into a borrowed string slice.\n///\n/// # Safety\n///\n/// - Assumes `ptr` is a valid, null-terminated UTF-8 C string pointer.\n/// - The returned `&str` borrows the underlying allocation; callers must ensure the\n///   C buffer outlives every use of the string slice.\n///\n/// # Panics\n///\n/// Panics if `ptr` is null or contains invalid UTF-8.\n#[must_use]\npub unsafe fn cstr_as_str<'a>(ptr: *const c_char) -> &'a str {\n    assert!(!ptr.is_null(), \"`ptr` was NULL\");\n    // SAFETY: Caller guarantees ptr is valid per function contract\n    let cstr = unsafe { CStr::from_ptr(ptr) };\n    cstr.to_str().expect(\"C string contains invalid UTF-8\")\n}\n\n/// Convert an optional C string pointer into `Option<&str>`.\n///\n/// # Safety\n///\n/// - Assumes `ptr` is a valid, null-terminated UTF-8 C string pointer or NULL.\n/// - Any borrowed string must not outlive the underlying allocation.\n///\n/// # Panics\n///\n/// Panics if `ptr` is not null but contains invalid UTF-8.\n#[must_use]\npub unsafe fn optional_cstr_to_str<'a>(ptr: *const c_char) -> Option<&'a str> {\n    if ptr.is_null() {\n        None\n    } else {\n        // SAFETY: Caller guarantees ptr is valid per function contract","sourceCodeStart":111,"sourceCodeEnd":147,"githubUrl":"https://github.com/nautechsystems/nautilus_trader/blob/18893faf8b356be3320add8de2f861b0b647cf06/crates/core/src/ffi/string.rs#L111-L147","documentation":"cstr_as_str in crates/core/src/ffi/string.rs:129 borrows a C string pointer as a Rust &str, asserting non-null and then requiring the bytes to be valid UTF-8 via cstr.to_str().expect(\"C string contains invalid UTF-8\"). It is the shared primitive behind optional_json_from_cstr, precision_from_cstr, min_increment_precision_from_cstr, and optional_cstr_to_str, so any FFI caller passing non-UTF-8 bytes panics here.","triggerScenarios":"Any call into optional_cstr_to_str / precision parsing / JSON parsing helpers with a pointer to non-UTF-8 bytes: legacy-encoded text (Latin-1, Shift-JIS), a truncated multi-byte sequence, or garbage bytes from an uninitialized buffer.","commonSituations":"Python 2-era or locale-dependent encodings leaking across the boundary; instrument/venue metadata from a vendor feed in a non-UTF-8 codepage; off-by-one buffer writes splitting a multi-byte character.","solutions":["Make the producer encode strictly UTF-8 before the FFI call.","Sanitize/re-encode the string at the boundary (Python: s.encode('utf-8', errors='replace') after decoding with the true source encoding).","Check buffer sizing/termination so multi-byte sequences are copied intact.","Hex-dump the pointed-to bytes to identify the encoding, then convert it explicitly to UTF-8 upstream."],"exampleFix":"// before\nvendor_name = raw_bytes.decode(\"latin-1\").encode(\"latin-1\")  # still non-UTF-8 bytes\n// after\nvendor_name = raw_bytes.decode(\"latin-1\").encode(\"utf-8\")   # valid UTF-8 for FFI","handlingStrategy":"validation","validationCode":"# Python caller\nb = s.encode(\"utf-8\")\nb.decode(\"utf-8\")                      # raises UnicodeDecodeError before crossing FFI","typeGuard":"fn is_valid_utf8(bytes: &[u8]) -> bool {\n    std::str::from_utf8(bytes).is_ok()\n}","tryCatchPattern":"// Caller-side pre-check when bytes are already in hand\nlet s = std::str::from_utf8(bytes)\n    .map_err(|_| \"non-UTF-8 bytes would panic in cstr_as_str\")?;","preventionTips":["Enforce UTF-8 at every serialization point; reject legacy encodings at ingest.","Round-trip encode/decode before crossing the boundary.","Hex-dump suspicious buffers when diagnosing to identify the actual encoding.","Include non-ASCII fixtures in integration tests."],"tags":["ffi","utf-8","encoding","panic"],"backgroundTag":"invalid-argument-format","analyzedSha":"18893faf8b356be3320add8de2f861b0b647cf06","analyzedAt":"2026-09-08T20:49:34.690Z","contentChangedAt":"2026-09-08T20:49:34.690Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}