rustfs/rustfs · error · std::io::Error

get object streaming body stall timeout

Error message

get object streaming body stall timeout

What it means

GetObjectStreamingReader wraps the object stream handed to the client with a stall watchdog: if no bytes are emitted for the configured timeout window, it fails the body with ErrorKind::TimedOut ('stall timeout') and logs expected/emitted/elapsed/timeout with state=stall_timeout. This protects the server from holding slots and buffers for clients (or intermediaries) that stop reading.

Source

Thrown at rustfs/src/app/object_usecase.rs:1782

            && std::future::Future::poll(timer.as_mut(), cx).is_ready()
        {
            self.timer = None;
            warn!(
                event = EVENT_GET_OBJECT_STREAM_BODY,
                component = LOG_COMPONENT_APP,
                subsystem = LOG_SUBSYSTEM_OBJECT,
                request_id = %self.request_id,
                range = %self.content_range.as_deref().unwrap_or("full"),
                size_bucket = get_object_stream_size_bucket(self.expected),
                expected = self.expected,
                emitted = self.emitted,
                elapsed_ms = self.elapsed().as_millis(),
                timeout_ms = self.timeout.as_millis(),
                state = "stall_timeout",
                "GetObject streaming body stalled"
            );
            self.finish_err();
            return Poll::Ready(Err(std::io::Error::new(
                std::io::ErrorKind::TimedOut,
                "get object streaming body stall timeout",
            )));
        }

        Poll::Pending
    }
}

impl<R: AsyncRead + Unpin> AsyncRead for GetObjectStreamingReader<R> {
    fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<std::io::Result<()>> {
        let filled_before = buf.filled().len();

        loop {
            // An armed resume owns the reader until it swaps in a reopened
            // stream or exhausts its budget; the failed inner stream is never
            // polled again.
            if self.resume_in_flight() {

View on GitHub (pinned to 9e6e02ea09)

Solutions

1. Make the client consume the body continuously (stream to disk, bounded buffers) instead of reading then pausing
2. Disable or tune whole-response buffering on any reverse proxy in front of the server
3. Raise the GET streaming stall timeout configuration if legitimately moving huge objects over slow links
4. Use ranged GETs / part-sized downloads so each request is short-lived and independently retryable

match body.read(...).await {
    Err(e) if e.kind() == std::io::ErrorKind::TimedOut
        && e.to_string().contains("stall timeout") => {
        // resume from the last byte received using a ranged GET
        resume_ranged_get(key, bytes_so_far).await
    }
    other => other,
}

When it happens

Trigger: A client that stops reading the response body (backpressure with no progress) for longer than the stall timeout; a proxy/CDN that buffers the entire object before forwarding; a network path that blackholes the connection without TCP failure.

Common situations: Download clients that process into a slow sink without reading the socket; nginx/traefik-style full-response buffering on large objects; mobile/flaky networks; aggressive firewall idle-timeouts between server and client.

Understand the failure class

• Timeouts: ETIMEDOUT, deadlines, and hung requests — what actually expires when a request times out.

Related errors

• IncompleteBody
• request body read stalled: received {} of {} bytes, no data
• unexpected GetBucketVersioning XML namespace
• host is none
• invalid X-Amz-Content-Sha256 header value: {err}

AI-assisted analysis of rustfs/rustfs@9e6e02ea09 (2026-08-16). Data as JSON: /api/errors/dcd6002146a4c2f9. Report an issue: GitHub.