{"record":{"id":"8789cd048bc655c7","repo":"influxdata/influxdb","slug":"error-converting-csv-output-to-utf-8","errorCode":null,"errorMessage":"Error converting CSV output to UTF-8: {}","messagePattern":"Error converting CSV output to UTF-8: (.+?)","errorType":"exception","errorClass":"Error","httpStatus":null,"severity":"error","filePath":"core/influxdb_iox_client/src/format.rs","lineNumber":34,"sourceCode":"pub enum Error {\n    /// Unknown formatting type\n    #[error(\"Unknown format type: {}. Expected one of 'pretty', 'csv' or 'json'\", .0)]\n    Invalid(String),\n\n    /// Error pretty printing\n    #[error(\"Arrow pretty printing error: {}\", .0)]\n    PrettyArrow(ArrowError),\n\n    /// Error during CSV conversion\n    #[error(\"Arrow csv printing error: {}\", .0)]\n    CsvArrow(ArrowError),\n\n    /// Error during JSON conversion\n    #[error(\"Arrow json printing error: {}\", .0)]\n    JsonArrow(ArrowError),\n\n    /// Error converting CSV output to utf-8\n    #[error(\"Error converting CSV output to UTF-8: {}\", .0)]\n    CsvUtf8(std::string::FromUtf8Error),\n\n    /// Error converting JSON output to utf-8\n    #[error(\"Error converting JSON output to UTF-8: {}\", .0)]\n    JsonUtf8(std::string::FromUtf8Error),\n}\ntype Result<T, E = Error> = std::result::Result<T, E>;\n\n#[derive(Debug, Copy, Clone, PartialEq, Eq, Default)]\n/// Requested output format for the query endpoint\npub enum QueryOutputFormat {\n    /// Arrow pretty printer format (default)\n    #[default]\n    Pretty,\n    /// Comma separated values\n    Csv,\n    /// Arrow JSON format\n    Json,","sourceCodeStart":16,"sourceCodeEnd":52,"githubUrl":"https://github.com/influxdata/influxdb/blob/06200ef96ba82c5f6727e5038a83af8e722c6875/core/influxdb_iox_client/src/format.rs#L16-L52","documentation":"This error is returned when the CSV byte output produced by the arrow CSV writer cannot be converted into a Rust UTF-8 String (std::string::FromUtf8Error). The library assumes CSV output is valid UTF-8; if not, it wraps the FromUtf8Error after 'Error converting CSV output to UTF-8: '.","triggerScenarios":"Calling format with Format::Csv when the underlying data contains byte sequences that decode to invalid UTF-8 in string columns.","commonSituations":"Ingested line-protocol or binary data with malformed encodings later surfaced through CSV export; data written by non-UTF-8 producers.","solutions":["Locate and clean the invalid UTF-8 source data at ingestion time.","Validate string columns are UTF-8 before querying/exporting.","Use the JSON or pretty format, which may surface data differently.","Decode bytes with from_utf8_lossy semantics in a custom export path."],"exampleFix":"// before\nlet csv = result.format(format::Format::Csv)?;\n// after\nlet csv = match result.format(format::Format::Csv) {\n    Ok(c) => c,\n    Err(format::Error::CsvUtf8(e)) => {\n        eprintln!(\"non-utf8 csv output: {e}\");\n        String::from_utf8_lossy(raw_bytes).into_owned()\n    }\n};","handlingStrategy":"validation","validationCode":"// Rust: verify string columns decode as UTF-8 before export\nfn strings_are_utf8(batches: &[RecordBatch]) -> bool {\n    batches.iter().all(|b| b.columns().iter().all(|c| {\n        c.data_type() != &DataType::Utf8 || {\n            let a = c.as_any().downcast_ref::<StringArray>().unwrap();\n            (0..a.len()).all(|i| a.is_null(i) || std::str::from_utf8(a.value(i)).is_ok())\n        }\n    }))\n}","typeGuard":"fn is_csv_utf8_err(e: &format::Error) -> Option<&std::string::FromUtf8Error> {\n    match e { format::Error::CsvUtf8(u) => Some(u), _ => None }\n}","tryCatchPattern":null,"preventionTips":["Sanitize ingested line-protocol string data for valid UTF-8 at write time","Reject or lossy-convert non-UTF-8 payloads at ingestion","Audit data sources that write raw bytes into string columns"],"tags":["utf-8","csv","encoding"],"backgroundTag":"invalid-argument-format","analyzedSha":"06200ef96ba82c5f6727e5038a83af8e722c6875","analyzedAt":"2026-09-19T12:55:30.003Z","contentChangedAt":"2026-09-19T12:55:30.003Z","schemaVersion":2},"datasetVersion":"2026-09-23T08:17:48.524Z"}