quickwit-oss/quickwit · error

Unexpected span kind: {}

Error message

Unexpected span kind: {}

What it means

get_or_open_split_file acquires a permit from an internal fd-limiting Semaphore before opening the split file. `Semaphore::acquire_owned` returns Err only when the semaphore has been closed. The `.expect` states this should never happen — the fd_semaphore is never closed in Quickwit — so it is an internal invariant: if you see this, the semaphore was closed while a fetch was in flight.

Source

Thrown at quickwit/quickwit-opentelemetry/src/otlp/traces.rs:343

        };
        Ok(span)
    }
}

#[derive(Debug, Clone)]
pub struct SpanKind(i32);

impl SpanKind {
    pub fn as_char(&self) -> char {
        match self.0 {
            0 => '0',
            1 => '1',
            2 => '2',
            3 => '3',
            4 => '4',
            5 => '5',
            _ => {
                panic!("Unexpected span kind: {}", self.0);
            }
        }
    }
    pub fn as_jaeger(&self) -> &'static str {
        match self.0 {
            0 => "unspecified",
            1 => "internal",
            2 => "server",
            3 => "client",
            4 => "producer",
            5 => "consumer",
            _ => {
                panic!("Unexpected span kind: {}", self.0);
            }
        }
    }

    pub fn as_otlp(&self) -> &'static str {

View on GitHub (pinned to a39730c5cd)

Solutions

1. Ensure the storage/file-descriptor cache is not dropped or shut down while fetch operations are still in flight — await pending fetches before teardown.
2. Check lifecycle ordering: the storage must outlive all searches/indexing tasks that reference it.
3. Reproduce with RUST_BACKTRACE=1 to find where the semaphore gets closed.
4. If it happens at shutdown only, tolerate it by treating a closed semaphore as a cancellation rather than a panic.

Example fix

// before
let fd_semaphore_guard = Semaphore::acquire_owned(self.fd_semaphore.clone())
    .await
    .expect("fd_semaphore acquire failed. please report");
// after
let fd_semaphore_guard = Semaphore::acquire_owned(self.fd_semaphore.clone())
    .await
    .map_err(|_| anyhow::anyhow!("fd_semaphore closed while acquiring split file; storage is shutting down"))?;

// at shutdown, wait for in-flight fetches before closing the storage
storage_shutdown_token.cancel().await;
tokio::time::timeout(Duration::from_secs(30), all_fetches.join())
    .await
    .expect("in-flight split fetches did not drain before semaphore close");

When it happens

Trigger: The fd semaphore is closed (via `semaphore.close()`) while get_or_open_split_file is waiting for a permit or about to acquire one — e.g. a shutdown/teardown path closing the semaphore while storage fetches are still running, or a bug in cache lifecycle management.

Common situations: Node shutdown or actor teardown racing with in-flight split downloads; tests (test_fd_cache_big_cache, test_split_file, etc.) that drop/close the cache while futures still run; storage objects being rebuilt while old ones are mid-fetch.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• `append_records` should be called with `position_opt: None`
• failed to run mrecordlog operation
• stdin cannot be checkpointed
• node not found in pending
• OTP logs or traces do not support VRL transforms

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/395a8dfe1d1b86b0. Report an issue: GitHub.