nautechsystems/nautilus_trader · error

C string contains invalid UTF-8

Error message

C string contains invalid UTF-8

What it means

cstr_as_str in crates/core/src/ffi/string.rs:129 borrows a C string pointer as a Rust &str, asserting non-null and then requiring the bytes to be valid UTF-8 via cstr.to_str().expect("C string contains invalid UTF-8"). It is the shared primitive behind optional_json_from_cstr, precision_from_cstr, min_increment_precision_from_cstr, and optional_cstr_to_str, so any FFI caller passing non-UTF-8 bytes panics here.

Source

Thrown at crates/core/src/ffi/string.rs:129

}

/// Convert a C string pointer into a borrowed string slice.
///
/// # Safety
///
/// - Assumes `ptr` is a valid, null-terminated UTF-8 C string pointer.
/// - The returned `&str` borrows the underlying allocation; callers must ensure the
///   C buffer outlives every use of the string slice.
///
/// # Panics
///
/// Panics if `ptr` is null or contains invalid UTF-8.
#[must_use]
pub unsafe fn cstr_as_str<'a>(ptr: *const c_char) -> &'a str {
    assert!(!ptr.is_null(), "`ptr` was NULL");
    // SAFETY: Caller guarantees ptr is valid per function contract
    let cstr = unsafe { CStr::from_ptr(ptr) };
    cstr.to_str().expect("C string contains invalid UTF-8")
}

/// Convert an optional C string pointer into `Option<&str>`.
///
/// # Safety
///
/// - Assumes `ptr` is a valid, null-terminated UTF-8 C string pointer or NULL.
/// - Any borrowed string must not outlive the underlying allocation.
///
/// # Panics
///
/// Panics if `ptr` is not null but contains invalid UTF-8.
#[must_use]
pub unsafe fn optional_cstr_to_str<'a>(ptr: *const c_char) -> Option<&'a str> {
    if ptr.is_null() {
        None
    } else {
        // SAFETY: Caller guarantees ptr is valid per function contract

View on GitHub (pinned to 18893faf8b)

Solutions

1. Make the producer encode strictly UTF-8 before the FFI call.
2. Sanitize/re-encode the string at the boundary (Python: s.encode('utf-8', errors='replace') after decoding with the true source encoding).
3. Check buffer sizing/termination so multi-byte sequences are copied intact.
4. Hex-dump the pointed-to bytes to identify the encoding, then convert it explicitly to UTF-8 upstream.

Example fix

// before
vendor_name = raw_bytes.decode("latin-1").encode("latin-1")  # still non-UTF-8 bytes
// after
vendor_name = raw_bytes.decode("latin-1").encode("utf-8")   # valid UTF-8 for FFI

# Python caller
b = s.encode("utf-8")
b.decode("utf-8")                      # raises UnicodeDecodeError before crossing FFI

fn is_valid_utf8(bytes: &[u8]) -> bool {
    std::str::from_utf8(bytes).is_ok()
}

// Caller-side pre-check when bytes are already in hand
let s = std::str::from_utf8(bytes)
    .map_err(|_| "non-UTF-8 bytes would panic in cstr_as_str")?;

When it happens

Trigger: Any call into optional_cstr_to_str / precision parsing / JSON parsing helpers with a pointer to non-UTF-8 bytes: legacy-encoded text (Latin-1, Shift-JIS), a truncated multi-byte sequence, or garbage bytes from an uninitialized buffer.

Common situations: Python 2-era or locale-dependent encodings leaking across the boundary; instrument/venue metadata from a vendor feed in a non-UTF-8 codepage; off-by-one buffer writes splitting a multi-byte character.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• CStr::from_ptr failed
• Invalid UTF-8 in C string
• Invalid scientific notation exponent '{exponent}': must be a
• invalid `AggressorSide` enum string value, was '{value}'
• invalid `AssetClass` enum string value, was '{value}'

AI-assisted analysis of nautechsystems/nautilus_trader@18893faf8b (2026-09-08). Data as JSON: /api/errors/3510494159714f62. Report an issue: GitHub.