What causes "HTTP agent request PUT {} failed {}" in windmill-labs/windmill?

Check server logs for the PUT request's error body (reason is not in the error message).

HTTP agent request PUT {} failed {}

Error message

HTTP agent request PUT {} failed {}

What it means

Raised by HttpClient::put_bytes when a worker's HTTP-agent PUT request uploading raw bytes returns a non-2xx status. No response body is preserved, so the status code in the message is the only signal of why the upload was rejected.

Source

Thrown at backend/windmill-common/src/worker.rs:642

                url,
                response.status()
            ))
        }
    }

    pub async fn put_bytes(&self, url: &str, bytes: Bytes) -> anyhow::Result<()> {
        let base_url = self.base_internal_url.clone();
        let response = self
            .client
            .put(format!("{}{}", base_url, url))
            .body(bytes)
            .send()
            .await
            .map_err(|e| anyhow::anyhow!(e))?;
        if response.status().is_success() {
            Ok(())
        } else {
            Err(anyhow::anyhow!(
                "HTTP agent request PUT {} failed {}",
                url,
                response.status()
            ))
        }
    }
}

#[derive(Clone)]
pub enum Connection {
    Sql(Pool<Postgres>),
    Http(HttpClient),
}

impl std::fmt::Debug for Connection {
    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
        match self {
            Connection::Sql(_) => write!(f, "Sql"),

View on GitHub (pinned to e474e8803c)

Solutions

Check server logs for the PUT request's error body (reason is not in the error message).
Verify storage backend health and configuration on the server (S3 creds, disk space).
Check the payload size against any upload limits and the token's write permissions.
Retry on 5xx with backoff; fix auth/size issues for 4xx.

Example fix

// before
client.put_bytes(&url, bytes).await?;
// after
if let Err(e) = client.put_bytes(&url, bytes.clone()).await {
    let msg = e.to_string();
    if msg.contains("500") || msg.contains("502") || msg.contains("503") {
        tokio::time::sleep(Duration::from_secs(3)).await;
        client.put_bytes(&url, bytes).await?;
    } else {
        return Err(e);
    }
}

Defensive patterns

Strategy: retry

Validate before calling

// Check size and payload sanity before PUT
if bytes.len() > MAX_UPLOAD_BYTES {
    return Err(anyhow::anyhow!("payload {} exceeds upload limit", bytes.len()));
}
if bytes.is_empty() {
    return Err(anyhow::anyhow!("refusing empty PUT payload"));
}

Try / catch

let mut attempt = 0;
loop {
    match client.put_bytes(&url, bytes.clone()).await {
        Ok(()) => break,
        Err(e) if attempt < 3 && (e.to_string().contains("500") || e.to_string().contains("502") || e.to_string().contains("503") || e.to_string().contains("504")) => {
            attempt += 1;
            tokio::time::sleep(Duration::from_secs(2u64.pow(attempt))).await;
        }
        Err(e) => return Err(anyhow::anyhow!("PUT {url} failed after {attempt} retries: {e}")),
    }
}

Prevention

Validate payload size against server upload limits before PUT.
Monitor storage backend health (S3 creds, disk space) that serves the upload endpoint.
Confirm the worker token has write permissions on the target resource.
Use idempotent uploads so retries are safe.
Route retries only on 5xx; surface 4xx (403/413) as config errors immediately.

When it happens

Trigger: Worker calls HttpClient::put_bytes to upload file/blob content and the server returns 4xx (auth, quota, invalid path, payload too large) or 5xx (storage backend failure).

Common situations: Storage backend (S3/MinIO/filesystem) down or misconfigured; payload exceeding size limits (413); write-permission denied for the worker token (403); expired storage credentials server-side; wrong base_internal_url.

Related errors

AI-assisted analysis of windmill-labs/windmill@e474e8803c (2026-09-03). Data as JSON: /api/errors/504460cdc211da4d. Report an issue: GitHub.