What causes "HTTP agent request POST {} failed {}" in windmill-labs/windmill?

Check the backend/API server logs for the matching request to get the real error body (the message only shows the status).

windmill-labs/windmill · error

HTTP agent request POST {} failed {}

Error message

HTTP agent request POST {} failed {}

What it means

Raised by HttpClient::post in windmill-common/src/worker.rs when a worker's HTTP-agent request (the Connection::Http path where a worker reaches the server API without direct DB access) returns a non-2xx status. The error message carries only the request path and the HTTP status code; the response body with the server's detailed error reason is discarded, so the status is all you get.

Source

Thrown at backend/windmill-common/src/worker.rs:583

    ) -> anyhow::Result<R> {
        let base_url = self.base_internal_url.clone();

        let response_builder = self.client.post(format!("{}{}", base_url, url)).json(body);

        let response_builder = match headers {
            Some(headers) => response_builder.headers(headers),
            None => response_builder,
        };

        let response = response_builder
            .send()
            .await
            .map_err(|e| anyhow::anyhow!(e))?;
        let status = response.status();
        if status.is_success() {
            Ok(response.json().await?)
        } else {
            Err(anyhow::anyhow!(format!(
                "HTTP agent request POST {} failed {}",
                url,
                response.status()
            )))
        }
    }

    pub async fn get<R: DeserializeOwned>(&self, url: &str) -> anyhow::Result<R> {
        let base_url = self.base_internal_url.clone();
        let response = self
            .client
            .get(format!("{}{}", base_url, url))
            .send()
            .await
            .map_err(|e| anyhow::anyhow!(e))?;
        let status = response.status();
        if status.is_success() {
            Ok(response.json().await?)

View on GitHub (pinned to e474e8803c)

Solutions

Check the backend/API server logs for the matching request to get the real error body (the message only shows the status).
Verify the worker's base_internal_url / BASE_URL points at the correct reachable server.
Confirm the worker's token has the required workspace permissions for the endpoint.
Retry on 5xx statuses (transient server errors); investigate request payload if 4xx.

Example fix

// before
let res: R = client.post(&url, headers, &body).await?;
// after
let res: R = match client.post(&url, headers, &body).await {
    Ok(r) => r,
    Err(e) if e.to_string().contains("500") || e.to_string().contains("502") || e.to_string().contains("503") => {
        tokio::time::sleep(Duration::from_secs(2)).await;
        client.post(&url, headers, &body).await?
    }
    Err(e) => return Err(e),
};

Defensive patterns

Strategy: try-catch

Validate before calling

// Rust: preflight reachability/health before the POST
if let Err(e) = client.get::<serde_json::Value>("/health").await {
    return Err(anyhow::anyhow!("server unreachable before POST {url}: {e}"));
}

Try / catch

match client.post::<_, R>(&url, headers, &body).await {
    Ok(r) => r,
    Err(e) => {
        let msg = e.to_string();
        if msg.contains("401") || msg.contains("403") {
            return Err(anyhow::anyhow!("auth/permission failure on POST {url}: {msg}"));
        }
        if msg.contains("500") || msg.contains("502") || msg.contains("503") || msg.contains("504") {
            // retry transient server errors with backoff
            tokio::time::sleep(Duration::from_secs(2)).await;
            client.post::<_, R>(&url, headers, &body).await.map_err(|e| anyhow::anyhow!("POST {url} retry failed: {e}"))?
        } else {
            return Err(anyhow::anyhow!("POST {url} rejected: {msg}"));
        }
    }
}

Prevention

Verify base_internal_url/BASE_URL during worker startup with a health check.
Grant the worker token only the permissions it needs, and test them before production jobs.
Add retry-with-backoff only for 5xx-class statuses, not 4xx.
Monitor server logs alongside worker logs so the discarded response body is visible.
Pin the server version/features the worker expects (feature-gated endpoints 404).

When it happens

Trigger: Any worker-side POST through HttpClient::post (workspace service calls, global service calls, OpenAPI service calls, mock AI API, unauthed service calls, route helpers) where the server responds 4xx or 5xx: invalid token/permissions, nonexistent resource, malformed body, server error, or wrong base_internal_url hitting an unexpected endpoint.

Common situations: Misconfigured BASE_URL/base_internal_url pointing at the wrong server; worker token lacking workspace permissions; server unreachable behind a proxy returning 404/502; calling an endpoint gated behind an enterprise feature or disabled on the target instance; transient 5xx during backend restarts.

Related errors

AI-assisted analysis of windmill-labs/windmill@e474e8803c (2026-09-03). Data as JSON: /api/errors/83e71ad335a1c8ba. Report an issue: GitHub.