{"record":{"id":"605a68c8ebd3097f","repo":"rustfs/rustfs","slug":"sidecar-runtime-queue-depth-queue-depth-exceeds","errorCode":null,"errorMessage":"sidecar runtime queue depth {queue_depth} exceeds policy bound {max_queue_depth}","messagePattern":"sidecar runtime queue depth (.+?) exceeds policy bound (.+?)","errorType":"exception","errorClass":"SidecarRuntimePolicyError","httpStatus":null,"severity":"error","filePath":"crates/targets/src/runtime/sidecar.rs","lineNumber":98,"sourceCode":"            sandboxed: true,\n            provenance_verified: true,\n            queue_depth,\n        }\n    }\n}\n\n#[derive(Debug, Error, PartialEq, Eq)]\npub enum SidecarRuntimePolicyError {\n    #[error(\"external sidecar runtime is disabled by policy\")]\n    ExternalSidecarDisabled,\n\n    #[error(\"sidecar runtime requires sandbox isolation\")]\n    SandboxRequired,\n\n    #[error(\"sidecar runtime requires verified provenance\")]\n    ProvenanceRequired,\n\n    #[error(\"sidecar runtime queue depth {queue_depth} exceeds policy bound {max_queue_depth}\")]\n    QueueDepthExceeded { queue_depth: usize, max_queue_depth: usize },\n}\n\n#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]\n#[serde(rename_all = \"snake_case\")]\npub struct SidecarPluginRuntime {\n    pub endpoint: String,\n    pub handshake: SidecarHandshake,\n    pub healthy: bool,\n    pub failure_count: usize,\n    pub degraded_to_builtin: bool,\n    pub last_error: Option<String>,\n}\n\nimpl SidecarPluginRuntime {\n    pub fn new(endpoint: impl Into<String>, handshake: SidecarHandshake) -> Self {\n        Self {\n            endpoint: endpoint.into(),","sourceCodeStart":80,"sourceCodeEnd":116,"githubUrl":"https://github.com/rustfs/rustfs/blob/35af688cd9d41b4346fbe27dcf7250ba72046c1f/crates/targets/src/runtime/sidecar.rs#L80-L116","documentation":"Returned by validate_runtime_policy (crates/targets/src/runtime/sidecar.rs:237-242) when safety_checks.queue_depth exceeds policy.max_queue_depth. Note the Default policy sets max_queue_depth to 0 (sidecar.rs:40), so any queued work at activation time fails unless the policy was sized via SidecarRuntimePolicy::verified_external.","triggerScenarios":"enable_with_policy / validate_activation where the reported queue_depth is greater than the policy bound - including a default policy (bound 0) or an undersized verified_external bound while the sidecar queue is backed up.","commonSituations":"Sidecar slower than the producer so pending work accumulates before activation; policy sized for steady state but activation happens during a burst; default policy reused without setting a queue budget.","solutions":["Drain or apply backpressure to the pending queue, then retry activation with a lower queue_depth","Size max_queue_depth to the real worst-case backlog when constructing the policy with SidecarRuntimePolicy::verified_external(max_queue_depth, ...)","If overruns recur, raise the bound deliberately and add monitoring - do not loop retries against a full queue"],"exampleFix":"// before\nlet policy = SidecarRuntimePolicy::verified_external(4, Duration::from_secs(5), 3);\nruntime.enable_with_policy(plugin_id, domain, &policy, &SidecarRuntimeSafetyChecks::verified(16))?; // 16 > 4\n\n// after\nlet policy = SidecarRuntimePolicy::verified_external(64, Duration::from_secs(5), 3);\nruntime.enable_with_policy(plugin_id, domain, &policy, &SidecarRuntimeSafetyChecks::verified(16))?;","handlingStrategy":"retry","validationCode":"fn within_queue_budget(policy: &SidecarRuntimePolicy, checks: &SidecarRuntimeSafetyChecks) -> bool {\n    checks.queue_depth <= policy.max_queue_depth\n}\n\nif !within_queue_budget(&policy, &checks) {\n    drain_or_backpressure(&mut queue, policy.max_queue_depth).await;\n}\nlet checks = SidecarRuntimeSafetyChecks::verified(queue.pending());\nruntime.enable_with_policy(plugin_id, domain, &policy, &checks)?;","typeGuard":"fn within_queue_budget(policy: &SidecarRuntimePolicy, checks: &SidecarRuntimeSafetyChecks) -> bool {\n    checks.queue_depth <= policy.max_queue_depth\n}","tryCatchPattern":"match runtime.enable_with_policy(plugin_id, domain, &policy, &checks) {\n    Err(e) if e.contains(\"exceeds policy bound\") => {\n        apply_backpressure(&mut queue, policy.max_queue_depth).await;\n        let checks = SidecarRuntimeSafetyChecks::verified(queue.pending());\n        runtime.enable_with_policy(plugin_id, domain, &policy, &checks)\n    }\n    other => other.map_err(Into::into),\n}","preventionTips":["Size max_queue_depth from measured worst-case backlog, not steady state","Report queue_depth in metrics so overruns are visible before activation fails","Remember the Default policy has max_queue_depth = 0: always construct the policy explicitly for external sidecars"],"tags":["rust","sidecar","backpressure","queue","capacity-planning"],"backgroundTag":"queue-depth-exceeded","analyzedSha":"35af688cd9d41b4346fbe27dcf7250ba72046c1f","analyzedAt":"2026-08-20T21:57:04.799Z","contentChangedAt":"2026-08-20T21:57:04.799Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}