{"record":{"id":"d8b63187575a2795","repo":"t8y2/dbx","slug":"invalid-byte-sequence-for-encoding","errorCode":null,"errorMessage":"Invalid byte sequence for {} encoding","messagePattern":"Invalid byte sequence for (.+?) encoding","errorType":"error_code","errorClass":"std::io::Error","httpStatus":null,"severity":"error","filePath":"crates/dbx-core/src/table_import.rs","lineNumber":555,"sourceCode":"        Ok(Self {\n            reader,\n            decoder,\n            encoding,\n            pending_input: Vec::with_capacity(IMPORT_ENCODING_READ_CHUNK_BYTES),\n            pending_output: Vec::new(),\n            output_offset: 0,\n            reached_eof: false,\n            finished: false,\n            source_bytes_read: 0,\n        })\n    }\n\n    fn source_bytes_read(&self) -> u64 {\n        self.source_bytes_read\n    }\n\n    fn invalid_data_error(&self) -> std::io::Error {\n        std::io::Error::new(\n            std::io::ErrorKind::InvalidData,\n            format!(\"Invalid byte sequence for {} encoding\", self.encoding.label()),\n        )\n    }\n}\n\nimpl<R: IoRead> IoRead for StrictTranscodingReader<R> {\n    fn read(&mut self, buffer: &mut [u8]) -> std::io::Result<usize> {\n        if buffer.is_empty() {\n            return Ok(0);\n        }\n\n        loop {\n            if self.output_offset < self.pending_output.len() {\n                let available = &self.pending_output[self.output_offset..];\n                let copied = available.len().min(buffer.len());\n                buffer[..copied].copy_from_slice(&available[..copied]);\n                self.output_offset += copied;","sourceCodeStart":537,"sourceCodeEnd":573,"githubUrl":"https://github.com/t8y2/dbx/blob/c0390bff16418b651f4728520d99adf8ce48829a/crates/dbx-core/src/table_import.rs#L537-L573","documentation":"The table-import pipeline's invalid_data_error constructs an std::io::Error of kind InvalidData with the message 'Invalid byte sequence for {encoding} encoding', where encoding.label() names the source encoding. It is raised when the importer reads bytes that cannot be decoded in the declared source encoding of the imported file.","triggerScenarios":"Importing a CSV/TSV/file whose bytes are not valid for the configured source encoding — e.g. a file containing raw UTF-8 multibyte sequences while the importer is told the encoding is latin-1/ascii, or truncated multibyte characters at chunk boundaries.","commonSituations":"Excel-exported CSVs in Windows-1252 labeled as UTF-8 (or vice versa), files concatenated from mixed encodings, byte-level corruption/truncation during transfer, or guessing the wrong encoding for legacy database dumps.","solutions":["Detect the file's real encoding (e.g. with a BOM check or chardet-style tool) and set the importer's source encoding to match.","Convert the file to UTF-8 with iconv or equivalent before importing.","Open and inspect the file around the reported byte offset to identify the offending bytes.","Re-export the source data ensuring a consistent, explicitly declared encoding."],"exampleFix":"// before\nimporter.set_source_encoding(\"utf-8\"); // file is actually Windows-1252\n// after\n// convert first: iconv -f WINDOWS-1252 -t UTF-8 input.csv > input.utf8.csv\nimporter.set_source_encoding(\"utf-8\");","handlingStrategy":"validation","validationCode":"// Rust\n// Detect/verify encoding before import:\nfn looks_like_utf8(bytes: &[u8]) -> bool {\n    std::str::from_utf8(bytes).is_ok()\n}\n// or use a detection crate (e.g. chardetng) and set the importer encoding accordingly","typeGuard":null,"tryCatchPattern":"// Rust\nmatch result {\n    Err(e) if e.kind() == std::io::ErrorKind::InvalidData\n        && e.to_string().contains(\"Invalid byte sequence for\") => {\n        // re-detect encoding, transcode the file to UTF-8, and retry the import\n    }\n    other => other,\n}","preventionTips":["Always convert source files to UTF-8 before importing.","Check for a BOM and run a detection pass instead of assuming the encoding.","Verify file integrity (no truncation) after transfers; inspect bytes near failure offsets."],"tags":["encoding","data-import","io"],"backgroundTag":"invalid-byte-sequence-encoding","analyzedSha":"c0390bff16418b651f4728520d99adf8ce48829a","analyzedAt":"2026-09-05T23:05:10.900Z","contentChangedAt":"2026-09-05T23:05:10.900Z","schemaVersion":2},"datasetVersion":"2026-09-14T05:17:10.506Z"}