What causes "Expected UTF-8." in vectordotdev/vector?

Fix the Lua script to emit valid UTF-8: hex- or base64-encode binary data before assigning it to a field

Expected UTF-8.

Error message

Expected UTF-8.

What it means

In the v1 Lua transform, assigning a string field with `event["field"] = value` converts the Lua string through `to_str().expect("Expected UTF-8.")`. Lua strings are raw byte arrays and mlua's `to_str()` errors on non-UTF-8 bytes, so assigning any binary string to an event field panics the transform and the pipeline task.

Source

Thrown at src/transforms/lua/v1/mod.rs:233

                    }
                })
            })
            .flatten(),
        )
    }
}

impl mlua::UserData for LuaEvent {
    fn add_methods<M: mlua::UserDataMethods<Self>>(methods: &mut M) {
        methods.add_meta_method_mut(
            mlua::MetaMethod::NewIndex,
            |_lua, this, (key, value): (String, Option<mlua::Value>)| {
                let key_path = parse_target_path(key.as_str()).map_err(|e| e.into_lua_err())?;
                match value {
                    Some(mlua::Value::String(string)) => {
                        this.inner.as_mut_log().insert(
                            &key_path,
                            Value::from(string.to_str().expect("Expected UTF-8.").to_owned()),
                        );
                    }
                    Some(mlua::Value::Integer(integer)) => {
                        this.inner
                            .as_mut_log()
                            .insert(&key_path, Value::Integer(integer));
                    }
                    Some(mlua::Value::Number(number)) if !number.is_nan() => {
                        this.inner
                            .as_mut_log()
                            .insert(&key_path, Value::Float(NotNan::new(number).unwrap()));
                    }
                    Some(mlua::Value::Boolean(boolean)) => {
                        this.inner
                            .as_mut_log()
                            .insert(&key_path, Value::Boolean(boolean));
                    }
                    Some(mlua::Value::Nil) | None => {

View on GitHub (pinned to 3708c39b12)

Solutions

Fix the Lua script to emit valid UTF-8: hex- or base64-encode binary data before assigning it to a field
Migrate to the v2 Lua transform (`lua` v2) or a remap/VRL transform, which handle bad bytes as errors/lossy instead of panicking
Guard assignments: validate the value is UTF-8 before setting the field, or escape non-ASCII bytes

Example fix

-- before
event["payload"] = chunk  -- chunk may contain non-UTF-8 bytes -> panic
-- after
event["payload"] = chunk:gsub("[^\1-\127]", function(b)
  return string.format("\\x%02X", b:byte())
end)

Defensive patterns

Strategy: type-guard

Validate before calling

-- Lua: validate before assigning
event["payload"] = to_utf8(chunk)

Type guard

-- Lua guard for LuaEvent __newindex inputs
local function to_utf8(s)
  return (s:gsub("[^\1-\127]", function(b)
    return string.format("\\x%02X", b:byte())
  end))
end

Try / catch

-- Lua v1: quarantine suspect values instead of panicking the pipeline
local ok, err = pcall(function()
  event["payload"] = chunk
end)
if not ok then
  event["payload_error"] = tostring(err)
end

Prevention

Never assign raw binary bytes to event fields in Lua v1; hex/base64-encode first
Migrate v1 Lua transforms to lua v2 or remap (VRL), which fail per-event without panicking
Feed binary sources through decode transforms (e.g., base64_decode) before Lua processing

When it happens

Trigger: A Lua script builds a non-UTF-8 byte string — `string.char(0xff)`, hex/protobuf-decoded blobs, `string.rep("\xff", 8)`, bytes passed through from a binary payload — and assigns it via the __newindex metamethod (`event["x"] = value`).

Common situations: Lua transforms handling binary-ish data (compressed fragments, legacy single-byte encodings, length-prefixed wire formats); scripts ported from byte-oriented Lua environments where string handling assumes bytes, not UTF-8 text.

Related errors

AI-assisted analysis of vectordotdev/vector@3708c39b12 (2026-08-20). Data as JSON: /api/errors/f9ddba7714b858a1. Report an issue: GitHub.