risingwavelabs/risingwave · error · ValueEncodingError

Invalid UTF8 value encoding: {0}

Error message

Invalid UTF8 value encoding: {0}

What it means

`ValueEncodingError::InvalidUtf8` is thrown when decoding a value-encoded string/VARCHAR: the encoded byte slice is converted to a Rust String, and invalid UTF-8 bytes cause `FromUtf8Error`, wrapped by this error via `#[from]`. It indicates the serialized text is not valid UTF-8, i.e. corrupt or foreign-encoded data.

Source

Thrown at src/common/src/util/value_encoding/error.rs:21

// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
//     http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

use thiserror::Error;

#[derive(Error, Debug)]
pub enum ValueEncodingError {
    #[error("Invalid bool value encoding: {0}")]
    InvalidBoolEncoding(u8),
    #[error("Invalid UTF8 value encoding: {0}")]
    InvalidUtf8(#[from] std::string::FromUtf8Error),
    #[error("Invalid Date value encoding: days: {0}")]
    InvalidDateEncoding(i32),
    #[error("invalid Timestamp value encoding: secs: {0} nsecs: {1}")]
    InvalidTimestampEncoding(i64, u32),
    #[error("invalid Time value encoding: secs: {0} nano: {1}")]
    InvalidTimeEncoding(u32, u32),
    #[error("Invalid null tag value encoding: {0}")]
    InvalidTagEncoding(u8),
    #[error("Invalid jsonb encoding")]
    InvalidJsonbEncoding,
    #[error("Invalid variant encoding")]
    InvalidVariantEncoding,
    #[error("Invalid struct encoding: {0}")]
    InvalidStructEncoding(
        #[source]
        #[backtrace]
        crate::array::ArrayError,

View on GitHub (pinned to 6469eb736d)

Solutions

1. Ensure upstream data is UTF-8: transcode it (e.g. to UTF-8) before writing into string columns
2. Validate that the byte range being decoded is a proper value-encoded string (correct length prefix/offset)
3. Inspect the offending bytes in the message to identify the corruption source and re-produce/re-ingest the data
4. Check writer/reader version compatibility of the value encoding format

Example fix

// before
let raw = vec![0xff, 0xfe]; // invalid UTF-8, e.g. Latin-1 text
let s = String::try_from_value_encoding(&raw)?;
// after
let raw = "café".to_string().into_bytes(); // valid UTF-8
let s = String::try_from_value_encoding(&raw)?;

// validate UTF-8 before decoding as value-encoded string
fn is_valid_utf8_encoding(bytes: &[u8]) -> bool {
    std::str::from_utf8(bytes).is_ok()
}

match String::try_from_value_encoding(bytes) {
    Ok(s) => s,
    Err(ValueEncodingError::InvalidUtf8(e)) => {
        // fall back to lossy decoding or reject the record
        let lossy = String::from_utf8_lossy(bytes).into_owned();
        return Ok(lossy);
    }
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: Decoding a value-encoded VARCHAR whose bytes contain invalid UTF-8 sequences (e.g. raw Latin-1/binary bytes), decoding at a wrong offset so a length-prefixed string slice covers unrelated bytes, or feeding non-UTF-8 producer data into a value-encoded string column.

Common situations: Ingesting binary payloads (MessagePack, protobuf) directly as text, GBK/Windows-1252 encoded input from upstream systems, corrupted Kafka messages, misaligned manual slicing of value-encoded buffers.

Understand the failure class

Background: "Must be a positive integer", "Invalid value", "Unsupported": the invalid-argument-value error family, when a library rejects the value you pass — this error's family across 35 libraries.

• Parsing and encoding errors: unexpected token, malformed input — why parsers reject input and how to find the real culprit.

Related errors

• Invalid bool value encoding: {0}
• Invalid Date value encoding: days: {0}
• invalid Timestamp value encoding: secs: {0} nsecs: {1}
• invalid Time value encoding: secs: {0} nano: {1}
• Invalid null tag value encoding: {0}

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/0f534bbe50aeacd1. Report an issue: GitHub.