t8y2/dbx · error · std::io::Error

Invalid byte sequence for {} encoding

Error message

Invalid byte sequence for {} encoding

What it means

The table-import pipeline's invalid_data_error constructs an std::io::Error of kind InvalidData with the message 'Invalid byte sequence for {encoding} encoding', where encoding.label() names the source encoding. It is raised when the importer reads bytes that cannot be decoded in the declared source encoding of the imported file.

Source

Thrown at crates/dbx-core/src/table_import.rs:555

        Ok(Self {
            reader,
            decoder,
            encoding,
            pending_input: Vec::with_capacity(IMPORT_ENCODING_READ_CHUNK_BYTES),
            pending_output: Vec::new(),
            output_offset: 0,
            reached_eof: false,
            finished: false,
            source_bytes_read: 0,
        })
    }

    fn source_bytes_read(&self) -> u64 {
        self.source_bytes_read
    }

    fn invalid_data_error(&self) -> std::io::Error {
        std::io::Error::new(
            std::io::ErrorKind::InvalidData,
            format!("Invalid byte sequence for {} encoding", self.encoding.label()),
        )
    }
}

impl<R: IoRead> IoRead for StrictTranscodingReader<R> {
    fn read(&mut self, buffer: &mut [u8]) -> std::io::Result<usize> {
        if buffer.is_empty() {
            return Ok(0);
        }

        loop {
            if self.output_offset < self.pending_output.len() {
                let available = &self.pending_output[self.output_offset..];
                let copied = available.len().min(buffer.len());
                buffer[..copied].copy_from_slice(&available[..copied]);
                self.output_offset += copied;

View on GitHub (pinned to c0390bff16)

Solutions

1. Detect the file's real encoding (e.g. with a BOM check or chardet-style tool) and set the importer's source encoding to match.
2. Convert the file to UTF-8 with iconv or equivalent before importing.
3. Open and inspect the file around the reported byte offset to identify the offending bytes.
4. Re-export the source data ensuring a consistent, explicitly declared encoding.

Example fix

// before
importer.set_source_encoding("utf-8"); // file is actually Windows-1252
// after
// convert first: iconv -f WINDOWS-1252 -t UTF-8 input.csv > input.utf8.csv
importer.set_source_encoding("utf-8");

// Rust
// Detect/verify encoding before import:
fn looks_like_utf8(bytes: &[u8]) -> bool {
    std::str::from_utf8(bytes).is_ok()
}
// or use a detection crate (e.g. chardetng) and set the importer encoding accordingly

// Rust
match result {
    Err(e) if e.kind() == std::io::ErrorKind::InvalidData
        && e.to_string().contains("Invalid byte sequence for") => {
        // re-detect encoding, transcode the file to UTF-8, and retry the import
    }
    other => other,
}

When it happens

Trigger: Importing a CSV/TSV/file whose bytes are not valid for the configured source encoding — e.g. a file containing raw UTF-8 multibyte sequences while the importer is told the encoding is latin-1/ascii, or truncated multibyte characters at chunk boundaries.

Common situations: Excel-exported CSVs in Windows-1252 labeled as UTF-8 (or vice versa), files concatenated from mixed encodings, byte-level corruption/truncation during transfer, or guessing the wrong encoding for legacy database dumps.

Related errors

• Custom H2 JDBC JAR does not exist: " + path
• H2 MVStore file is too small to contain valid headers: " + f
• Failed to read CLOB value
• PE import name is not ASCII
• reader cannot read token payload

AI-assisted analysis of t8y2/dbx@c0390bff16 (2026-09-05). Data as JSON: /api/errors/d8b63187575a2795. Report an issue: GitHub.