{"record":{"id":"395a8dfe1d1b86b0","repo":"quickwit-oss/quickwit","slug":"unexpected-span-kind","errorCode":null,"errorMessage":"Unexpected span kind: {}","messagePattern":"Unexpected span kind: (.+?)","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"error","filePath":"quickwit/quickwit-opentelemetry/src/otlp/traces.rs","lineNumber":343,"sourceCode":"        };\n        Ok(span)\n    }\n}\n\n#[derive(Debug, Clone)]\npub struct SpanKind(i32);\n\nimpl SpanKind {\n    pub fn as_char(&self) -> char {\n        match self.0 {\n            0 => '0',\n            1 => '1',\n            2 => '2',\n            3 => '3',\n            4 => '4',\n            5 => '5',\n            _ => {\n                panic!(\"Unexpected span kind: {}\", self.0);\n            }\n        }\n    }\n    pub fn as_jaeger(&self) -> &'static str {\n        match self.0 {\n            0 => \"unspecified\",\n            1 => \"internal\",\n            2 => \"server\",\n            3 => \"client\",\n            4 => \"producer\",\n            5 => \"consumer\",\n            _ => {\n                panic!(\"Unexpected span kind: {}\", self.0);\n            }\n        }\n    }\n\n    pub fn as_otlp(&self) -> &'static str {","sourceCodeStart":325,"sourceCodeEnd":361,"githubUrl":"https://github.com/quickwit-oss/quickwit/blob/a39730c5cdcd1a4fe798403737ae293999ea21f8/quickwit/quickwit-opentelemetry/src/otlp/traces.rs#L325-L361","documentation":"get_or_open_split_file acquires a permit from an internal fd-limiting Semaphore before opening the split file. `Semaphore::acquire_owned` returns Err only when the semaphore has been closed. The `.expect` states this should never happen — the fd_semaphore is never closed in Quickwit — so it is an internal invariant: if you see this, the semaphore was closed while a fetch was in flight.","triggerScenarios":"The fd semaphore is closed (via `semaphore.close()`) while get_or_open_split_file is waiting for a permit or about to acquire one — e.g. a shutdown/teardown path closing the semaphore while storage fetches are still running, or a bug in cache lifecycle management.","commonSituations":"Node shutdown or actor teardown racing with in-flight split downloads; tests (test_fd_cache_big_cache, test_split_file, etc.) that drop/close the cache while futures still run; storage objects being rebuilt while old ones are mid-fetch.","solutions":["Ensure the storage/file-descriptor cache is not dropped or shut down while fetch operations are still in flight — await pending fetches before teardown.","Check lifecycle ordering: the storage must outlive all searches/indexing tasks that reference it.","Reproduce with RUST_BACKTRACE=1 to find where the semaphore gets closed.","If it happens at shutdown only, tolerate it by treating a closed semaphore as a cancellation rather than a panic."],"exampleFix":"// before\nlet fd_semaphore_guard = Semaphore::acquire_owned(self.fd_semaphore.clone())\n    .await\n    .expect(\"fd_semaphore acquire failed. please report\");\n// after\nlet fd_semaphore_guard = Semaphore::acquire_owned(self.fd_semaphore.clone())\n    .await\n    .map_err(|_| anyhow::anyhow!(\"fd_semaphore closed while acquiring split file; storage is shutting down\"))?;","handlingStrategy":"try-catch","validationCode":null,"typeGuard":null,"tryCatchPattern":"// at shutdown, wait for in-flight fetches before closing the storage\nstorage_shutdown_token.cancel().await;\ntokio::time::timeout(Duration::from_secs(30), all_fetches.join())\n    .await\n    .expect(\"in-flight split fetches did not drain before semaphore close\");","preventionTips":["Never close the fd semaphore while split fetches are in flight.","Drain active searches/indexing tasks before dropping storage instances.","Verify storage lifetime outlives all holders of SplitFile handles.","If seen only at shutdown, treat closed-semaphore as graceful cancellation."],"tags":["rust","tokio","semaphore","file-descriptor","cache","storage"],"backgroundTag":"internal-invariant-violation","analyzedSha":"a39730c5cdcd1a4fe798403737ae293999ea21f8","analyzedAt":"2026-09-08T13:19:37.784Z","contentChangedAt":"2026-09-08T13:19:37.784Z","schemaVersion":2},"datasetVersion":"2026-09-14T11:17:12.474Z"}