rustfs/rustfs · error · SidecarRuntimePolicyError

sidecar runtime queue depth {queue_depth} exceeds policy bou

Error message

sidecar runtime queue depth {queue_depth} exceeds policy bound {max_queue_depth}

What it means

Returned by validate_runtime_policy (crates/targets/src/runtime/sidecar.rs:237-242) when safety_checks.queue_depth exceeds policy.max_queue_depth. Note the Default policy sets max_queue_depth to 0 (sidecar.rs:40), so any queued work at activation time fails unless the policy was sized via SidecarRuntimePolicy::verified_external.

Source

Thrown at crates/targets/src/runtime/sidecar.rs:98

            sandboxed: true,
            provenance_verified: true,
            queue_depth,
        }
    }
}

#[derive(Debug, Error, PartialEq, Eq)]
pub enum SidecarRuntimePolicyError {
    #[error("external sidecar runtime is disabled by policy")]
    ExternalSidecarDisabled,

    #[error("sidecar runtime requires sandbox isolation")]
    SandboxRequired,

    #[error("sidecar runtime requires verified provenance")]
    ProvenanceRequired,

    #[error("sidecar runtime queue depth {queue_depth} exceeds policy bound {max_queue_depth}")]
    QueueDepthExceeded { queue_depth: usize, max_queue_depth: usize },
}

#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
#[serde(rename_all = "snake_case")]
pub struct SidecarPluginRuntime {
    pub endpoint: String,
    pub handshake: SidecarHandshake,
    pub healthy: bool,
    pub failure_count: usize,
    pub degraded_to_builtin: bool,
    pub last_error: Option<String>,
}

impl SidecarPluginRuntime {
    pub fn new(endpoint: impl Into<String>, handshake: SidecarHandshake) -> Self {
        Self {
            endpoint: endpoint.into(),

View on GitHub (pinned to 35af688cd9)

Solutions

1. Drain or apply backpressure to the pending queue, then retry activation with a lower queue_depth
2. Size max_queue_depth to the real worst-case backlog when constructing the policy with SidecarRuntimePolicy::verified_external(max_queue_depth, ...)
3. If overruns recur, raise the bound deliberately and add monitoring - do not loop retries against a full queue

Example fix

// before
let policy = SidecarRuntimePolicy::verified_external(4, Duration::from_secs(5), 3);
runtime.enable_with_policy(plugin_id, domain, &policy, &SidecarRuntimeSafetyChecks::verified(16))?; // 16 > 4

// after
let policy = SidecarRuntimePolicy::verified_external(64, Duration::from_secs(5), 3);
runtime.enable_with_policy(plugin_id, domain, &policy, &SidecarRuntimeSafetyChecks::verified(16))?;

fn within_queue_budget(policy: &SidecarRuntimePolicy, checks: &SidecarRuntimeSafetyChecks) -> bool {
    checks.queue_depth <= policy.max_queue_depth
}

if !within_queue_budget(&policy, &checks) {
    drain_or_backpressure(&mut queue, policy.max_queue_depth).await;
}
let checks = SidecarRuntimeSafetyChecks::verified(queue.pending());
runtime.enable_with_policy(plugin_id, domain, &policy, &checks)?;

fn within_queue_budget(policy: &SidecarRuntimePolicy, checks: &SidecarRuntimeSafetyChecks) -> bool {
    checks.queue_depth <= policy.max_queue_depth
}

match runtime.enable_with_policy(plugin_id, domain, &policy, &checks) {
    Err(e) if e.contains("exceeds policy bound") => {
        apply_backpressure(&mut queue, policy.max_queue_depth).await;
        let checks = SidecarRuntimeSafetyChecks::verified(queue.pending());
        runtime.enable_with_policy(plugin_id, domain, &policy, &checks)
    }
    other => other.map_err(Into::into),
}

When it happens

Trigger: enable_with_policy / validate_activation where the reported queue_depth is greater than the policy bound - including a default policy (bound 0) or an undersized verified_external bound while the sidecar queue is backed up.

Common situations: Sidecar slower than the producer so pending work accumulates before activation; policy sized for steady state but activation happens during a burst; default policy reused without setting a queue budget.

Related errors

• Queue is full: {message}
• object data cache max_entry_bytes plus weigher overhead must
• object data cache min_free_memory_percent must be in 1..=100
• Entry limit exceeded
• Queued payload dropped: {0}

AI-assisted analysis of rustfs/rustfs@35af688cd9 (2026-08-20). Data as JSON: /api/errors/605a68c8ebd3097f. Report an issue: GitHub.