rustfs/rustfs · info

pull canceled

Error message

pull canceled

What it means

"pull canceled" is produced by PumpState::fail (rustfs/src/on_demand_migration/pull.rs:394) when the pull's CancellationToken fires while the source-body pump task is running. The pump records PumpFailure::Canceled in the shared failure slot and returns an io::Error with ErrorKind::Interrupted and this message, which propagates to the in-flight write-back body so the local upload fails promptly instead of hanging.

Source

Thrown at rustfs/src/on_demand_migration/pull.rs:394

}

impl PumpState {
    fn fail(&self, failure: PumpFailure) -> io::Error {
        let message = match &failure {
            PumpFailure::Source(err) => err.to_string(),
            PumpFailure::Canceled => "pull canceled".to_string(),
        };
        let kind = match &failure {
            PumpFailure::Source(SourceError::Timeout) => io::ErrorKind::TimedOut,
            PumpFailure::Source(SourceError::Connect(_)) => io::ErrorKind::UnexpectedEof,
            PumpFailure::Source(_) => io::ErrorKind::Other,
            PumpFailure::Canceled => io::ErrorKind::Interrupted,
        };
        let mut slot = self.failure.lock();
        if slot.is_none() {
            *slot = Some(failure);
        }
        io::Error::new(kind, message)
    }

    fn take(&self) -> Option<PumpFailure> {
        self.failure.lock().take()
    }
}

/// Copies the source body into a bounded channel, enforcing `idle_timeout`
/// per chunk (through [`idle_guarded_body`]), `cancel`, and the advertised
/// `expected_size`.
fn spawn_pump(
    body: SourceBody,
    expected_size: u64,
    idle_timeout: Duration,
    cancel: CancellationToken,
) -> (mpsc::Receiver<io::Result<Bytes>>, Arc<PumpState>) {
    let (mut body, _idle) = idle_guarded_body(body, idle_timeout);
    let (tx, rx) = mpsc::channel(PUMP_CHANNEL_CHUNKS);

View on GitHub (pinned to 5dca076efe)

Solutions

1. Treat this as a deliberate cancellation, not a data error: check whether the bucket's ODM config was changed or the node is shutting down at that time.
2. Re-trigger the pull (re-request the object) after the config settles; on-demand migration will pull again on the next access.
3. If cancellations are unexpected, audit who cancels the CancellationToken (shutdown hooks, config reload paths) and whether pulls are being restarted concurrently.
4. Do not retry inside the request handler: the failure slot holds PumpFailure::Canceled and the pipeline intentionally stops the write-back.

fn was_canceled(err: &io::Error) -> bool { err.kind() == io::ErrorKind::Interrupted && err.to_string() == "pull canceled" }

match pull_outcome {
    Err(err) if err.kind() == io::ErrorKind::Interrupted => {
        // cancellation, not corruption: no cleanup beyond pipeline's own abort
        tracing::debug!("pull canceled before commit");
    }
    Err(err) => record_pull_failure(err),
    Ok(c) => handle(c),
}

When it happens

Trigger: The cancellation token passed to spawn_pump is cancelled during an active pull: the ODM pull task is being shut down (server shutdown, bucket ODM disabled), the pull was superseded, or an explicit cancel/abort path fires while body chunks are still being streamed from the source.

Common situations: RustFS shutdown or reload of the on-demand-migration config while large objects are mid-pull; an operator disabling ODM for a bucket; a pull racing with its own retry/timeout logic that cancelled the token.

Related errors

• erasure reader fill worker request queue is closed or full
• unexpected header
• invalid encryption algorithm ID: {0}
• crypto feature is disabled
• erasure reader source missing

AI-assisted analysis of rustfs/rustfs@5dca076efe (2026-09-06). Data as JSON: /api/errors/70fd8bd3bd4ab6e2. Report an issue: GitHub.