{"record":{"id":"70fd8bd3bd4ab6e2","repo":"rustfs/rustfs","slug":"pull-canceled","errorCode":null,"errorMessage":"pull canceled","messagePattern":"pull canceled","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"info","filePath":"rustfs/src/on_demand_migration/pull.rs","lineNumber":394,"sourceCode":"}\n\nimpl PumpState {\n    fn fail(&self, failure: PumpFailure) -> io::Error {\n        let message = match &failure {\n            PumpFailure::Source(err) => err.to_string(),\n            PumpFailure::Canceled => \"pull canceled\".to_string(),\n        };\n        let kind = match &failure {\n            PumpFailure::Source(SourceError::Timeout) => io::ErrorKind::TimedOut,\n            PumpFailure::Source(SourceError::Connect(_)) => io::ErrorKind::UnexpectedEof,\n            PumpFailure::Source(_) => io::ErrorKind::Other,\n            PumpFailure::Canceled => io::ErrorKind::Interrupted,\n        };\n        let mut slot = self.failure.lock();\n        if slot.is_none() {\n            *slot = Some(failure);\n        }\n        io::Error::new(kind, message)\n    }\n\n    fn take(&self) -> Option<PumpFailure> {\n        self.failure.lock().take()\n    }\n}\n\n/// Copies the source body into a bounded channel, enforcing `idle_timeout`\n/// per chunk (through [`idle_guarded_body`]), `cancel`, and the advertised\n/// `expected_size`.\nfn spawn_pump(\n    body: SourceBody,\n    expected_size: u64,\n    idle_timeout: Duration,\n    cancel: CancellationToken,\n) -> (mpsc::Receiver<io::Result<Bytes>>, Arc<PumpState>) {\n    let (mut body, _idle) = idle_guarded_body(body, idle_timeout);\n    let (tx, rx) = mpsc::channel(PUMP_CHANNEL_CHUNKS);","sourceCodeStart":376,"sourceCodeEnd":412,"githubUrl":"https://github.com/rustfs/rustfs/blob/5dca076efed96e7b842de07c4c2111035ae7c7a2/rustfs/src/on_demand_migration/pull.rs#L376-L412","documentation":"\"pull canceled\" is produced by PumpState::fail (rustfs/src/on_demand_migration/pull.rs:394) when the pull's CancellationToken fires while the source-body pump task is running. The pump records PumpFailure::Canceled in the shared failure slot and returns an io::Error with ErrorKind::Interrupted and this message, which propagates to the in-flight write-back body so the local upload fails promptly instead of hanging.","triggerScenarios":"The cancellation token passed to spawn_pump is cancelled during an active pull: the ODM pull task is being shut down (server shutdown, bucket ODM disabled), the pull was superseded, or an explicit cancel/abort path fires while body chunks are still being streamed from the source.","commonSituations":"RustFS shutdown or reload of the on-demand-migration config while large objects are mid-pull; an operator disabling ODM for a bucket; a pull racing with its own retry/timeout logic that cancelled the token.","solutions":["Treat this as a deliberate cancellation, not a data error: check whether the bucket's ODM config was changed or the node is shutting down at that time.","Re-trigger the pull (re-request the object) after the config settles; on-demand migration will pull again on the next access.","If cancellations are unexpected, audit who cancels the CancellationToken (shutdown hooks, config reload paths) and whether pulls are being restarted concurrently.","Do not retry inside the request handler: the failure slot holds PumpFailure::Canceled and the pipeline intentionally stops the write-back."],"exampleFix":null,"handlingStrategy":"try-catch","validationCode":null,"typeGuard":"fn was_canceled(err: &io::Error) -> bool { err.kind() == io::ErrorKind::Interrupted && err.to_string() == \"pull canceled\" }","tryCatchPattern":"match pull_outcome {\n    Err(err) if err.kind() == io::ErrorKind::Interrupted => {\n        // cancellation, not corruption: no cleanup beyond pipeline's own abort\n        tracing::debug!(\"pull canceled before commit\");\n    }\n    Err(err) => record_pull_failure(err),\n    Ok(c) => handle(c),\n}","preventionTips":["Avoid toggling bucket ODM config or shutting down while large pulls are active; drain in-flight pulls first.","Ensure only one pull attempt per key is scheduled so retries don't cancel each other's tokens.","Log cancellation events at debug/info to correlate with shutdown or config-reload timestamps."],"tags":["cancellation","migration","async","rust"],"backgroundTag":"operation-cancelled","analyzedSha":"5dca076efed96e7b842de07c4c2111035ae7c7a2","analyzedAt":"2026-09-06T05:54:05.891Z","contentChangedAt":"2026-09-06T05:54:05.891Z","schemaVersion":2},"datasetVersion":"2026-09-14T00:17:10.932Z"}