influxdata/influxdb · error · Error

Error converting CSV output to UTF-8

Error message

Error converting CSV output to UTF-8: {}

What it means

This error is returned when the CSV byte output produced by the arrow CSV writer cannot be converted into a Rust UTF-8 String (std::string::FromUtf8Error). The library assumes CSV output is valid UTF-8; if not, it wraps the FromUtf8Error after 'Error converting CSV output to UTF-8: '.

Solutions

1. Locate and clean the invalid UTF-8 source data at ingestion time.
2. Validate string columns are UTF-8 before querying/exporting.
3. Use the JSON or pretty format, which may surface data differently.
4. Decode bytes with from_utf8_lossy semantics in a custom export path.

Example fix

// before
let csv = result.format(format::Format::Csv)?;
// after
let csv = match result.format(format::Format::Csv) {
    Ok(c) => c,
    Err(format::Error::CsvUtf8(e)) => {
        eprintln!("non-utf8 csv output: {e}");
        String::from_utf8_lossy(raw_bytes).into_owned()
    }
};

// Rust: verify string columns decode as UTF-8 before export
fn strings_are_utf8(batches: &[RecordBatch]) -> bool {
    batches.iter().all(|b| b.columns().iter().all(|c| {
        c.data_type() != &DataType::Utf8 || {
            let a = c.as_any().downcast_ref::<StringArray>().unwrap();
            (0..a.len()).all(|i| a.is_null(i) || std::str::from_utf8(a.value(i)).is_ok())
        }
    }))
}

fn is_csv_utf8_err(e: &format::Error) -> Option<&std::string::FromUtf8Error> {
    match e { format::Error::CsvUtf8(u) => Some(u), _ => None }
}

When it happens

Trigger: Calling format with Format::Csv when the underlying data contains byte sequences that decode to invalid UTF-8 in string columns.

Common situations: Ingested line-protocol or binary data with malformed encodings later surfaced through CSV export; data written by non-UTF-8 producers.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• Error converting JSON output to UTF-8
• Arrow csv printing error
• body content is not valid utf8
• from hex error
• invalid content-encoding header

AI-assisted analysis of influxdata/influxdb@06200ef96b (2026-09-19). Data as JSON: /api/errors/8789cd048bc655c7. Report an issue: GitHub.

Appendix: source

Thrown at core/influxdb_iox_client/src/format.rs:34

pub enum Error {
    /// Unknown formatting type
    #[error("Unknown format type: {}. Expected one of 'pretty', 'csv' or 'json'", .0)]
    Invalid(String),

    /// Error pretty printing
    #[error("Arrow pretty printing error: {}", .0)]
    PrettyArrow(ArrowError),

    /// Error during CSV conversion
    #[error("Arrow csv printing error: {}", .0)]
    CsvArrow(ArrowError),

    /// Error during JSON conversion
    #[error("Arrow json printing error: {}", .0)]
    JsonArrow(ArrowError),

    /// Error converting CSV output to utf-8
    #[error("Error converting CSV output to UTF-8: {}", .0)]
    CsvUtf8(std::string::FromUtf8Error),

    /// Error converting JSON output to utf-8
    #[error("Error converting JSON output to UTF-8: {}", .0)]
    JsonUtf8(std::string::FromUtf8Error),
}
type Result<T, E = Error> = std::result::Result<T, E>;

#[derive(Debug, Copy, Clone, PartialEq, Eq, Default)]
/// Requested output format for the query endpoint
pub enum QueryOutputFormat {
    /// Arrow pretty printer format (default)
    #[default]
    Pretty,
    /// Comma separated values
    Csv,
    /// Arrow JSON format
    Json,

View on GitHub (pinned to 06200ef96b)