databendlabs/databend · error

input value is not valid utf-8

Error message

input value is not valid utf-8: {err:?}

What it means

TomlIgnored::parse expects the input bytes to be valid UTF-8 before deserializing TOML with serde_ignored (to report ignored keys via the handler). If the byte slice is not valid UTF-8 it fails with "input value is not valid utf-8: {err:?}". TOML is a text format, so binary input cannot be parsed.

Solutions

1. Re-save the config file as UTF-8 (strip BOM; in vim: :set fileencoding=utf-8 and :set bomb<)
2. Check the file encoding with `file config.toml` and convert via iconv if needed
3. Verify the bytes passed to parse come from the intended text config, not a binary file
4. If the config came from a remote source, re-download and confirm content-type/text integrity

Example fix

// before: raw bytes straight into TOML parser
let cfg: Config = TomlIgnored::default().parse(&raw_bytes)?;
// after: validate/convert encoding first
let text = String::from_utf8(raw_bytes)
    .map_err(|e| anyhow!("config must be UTF-8: {e}"))?;
let cfg: Config = TomlIgnored::default().parse(text.as_bytes())?;

std::str::from_utf8(bs).map_err(|e| anyhow!("config bytes must be UTF-8: {e}"))?;

fn is_utf8(bs: &[u8]) -> bool { std::str::from_utf8(bs).is_ok() }

match TomlIgnored::default().parse::<Config>(bs) {
    Err(e) if e.to_string().contains("not valid utf-8") => bail!("re-save config as UTF-8"),
    other => other,
}

When it happens

Trigger: Calling parse::<T>(bs) on TomlIgnored where bs contains non-UTF-8 bytes — e.g. a config file with binary content, a wrong-encoding file (UTF-16, Latin-1), or bytes read from the wrong source.

Common situations: Config files saved by editors with BOM/UTF-16 encoding; config accidentally fetched from a binary endpoint; concatenated/corrupted config bytes.

Understand the failure class

Background: "Invalid ... format", "must be in format X", "does not look like a ..." — invalid argument format errors across CLI tools and libraries — this error's family across 17 libraries.

Related errors

• {msg}
• Invalid config.rpc
• not implemented: Not implemented for storage type
• grpc_api_address MUST not be empty!
• remove_percent MUST in [0, 100)!

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/4da146ca141a80b7. Report an issue: GitHub.

Appendix: source

Thrown at src/query/config/src/toml.rs:44

type TomlUnknownFieldHandler = Box<dyn Fn(&str) + Send + Sync + 'static>;

impl TomlIgnored {
    pub fn new(handler: TomlUnknownFieldHandler) -> Self {
        Self { handler }
    }
}

impl std::fmt::Debug for TomlIgnored {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        f.debug_struct("TomlIgnored").finish()
    }
}

impl Parser for TomlIgnored {
    fn parse<T: DeserializeOwned>(&mut self, bs: &[u8]) -> Result<T> {
        let s = std::str::from_utf8(bs)
            .map_err(|err| anyhow!("input value is not valid utf-8: {err:?}"))?;
        let de = toml::Deserializer::new(s);
        let handler = &self.handler;
        Ok(serde_ignored::deserialize(de, move |path| {
            handler(path.to_string().as_str());
        })?)
    }
}

View on GitHub (pinned to 288d84d76e)