quickwit-oss/quickwit · critical

OTP logs or traces do not support VRL transforms

Error message

OTP logs or traces do not support VRL transforms

What it means

Same pattern as the REST server: the health check server runs as a named Tokio task and its JoinHandle is awaited. `.expect` fires when that task panics or is cancelled, which the library treats as unrecoverable — the health endpoint must stay alive for the node to be considered healthy.

Source

Thrown at quickwit/quickwit-indexing/src/actors/doc_processor.rs:145

fn try_into_vrl_doc(
    input_format: SourceInputFormat,
    raw_doc: Bytes,
    num_bytes: usize,
) -> Result<VrlDoc, DocProcessorError> {
    let vrl_value = match input_format {
        SourceInputFormat::Json => serde_json::from_slice::<VrlValue>(&raw_doc)?,
        SourceInputFormat::PlainText => {
            let mut map = std::collections::BTreeMap::new();
            let key = vrl::value::KeyString::from(PLAIN_TEXT);
            let value = VrlValue::Bytes(raw_doc);
            map.insert(key, value);
            VrlValue::Object(map)
        }
        SourceInputFormat::OtlpLogsJson
        | SourceInputFormat::OtlpLogsProtobuf
        | SourceInputFormat::OtlpTracesJson
        | SourceInputFormat::OtlpTracesProtobuf => {
            panic!("OTP logs or traces do not support VRL transforms")
        }
    };
    let vrl_doc = VrlDoc::new(vrl_value, num_bytes);
    Ok(vrl_doc)
}

fn try_into_json_docs(
    input_format: SourceInputFormat,
    raw_doc: Bytes,
    num_bytes: usize,
) -> JsonDocIterator {
    match input_format {
        SourceInputFormat::Json => {
            let json_doc_result = serde_json::from_slice::<JsonObject>(&raw_doc)
                .map(|json_obj| JsonDoc::new(json_obj, num_bytes));
            JsonDocIterator::from(json_doc_result)
        }
        SourceInputFormat::OtlpLogsJson => {

View on GitHub (pinned to a39730c5cd)

Solutions

1. Look at the log lines preceding the panic to identify the actual panic in the health server task and fix it.
2. Reproduce with RUST_BACKTRACE=full to get the panic location.
3. Ensure the health-check port is free and not duplicated in the config so startup errors surface as clean errors instead of task panics.
4. Check host memory/limits; restart with adequate resources if the task was killed under pressure.

Example fix

// before
.expect("tasks running the health check server should not panic or be cancelled")
// after
.map_err(|err| anyhow::anyhow!("health check server task failed: {err}"))
.and_then(|res| res.context("health check server failed"))

// treat node exit as fatal and capture cause
if let Err(err) = serve_quickwit(config).await {
    tracing::error!(error = ?err, "server terminated");
    std::process::exit(1);
}

When it happens

Trigger: The `health_server_fut` panics (e.g. failure binding the health port is handled upstream, but a panic inside readiness/liveness state access does not); or the task is cancelled by runtime shutdown.

Common situations: Port already bound by another process after an earlier misconfigured start (leading to an awaited error path being mishandled), panic inside health handlers accessing cluster state, OOM conditions, or running the binary under a supervisor that kills tasks mid-flight.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• node not found in pending
• position of a Kafka partition should never be EOF
• position of a Kinesis shard should never be EOF
• `doc_batch` should not be empty
• Unexpected span kind: {}

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/a617601c4caba8a9. Report an issue: GitHub.