What causes "HTTP agent request GET {} failed {}" in windmill-labs/windmill?

Check server logs for the corresponding GET request to see the discarded error body.

HTTP agent request GET {} failed {}

Error message

HTTP agent request GET {} failed {}

What it means

Raised by HttpClient::get when a worker's HTTP-agent GET request returns a non-2xx status. Called by get_ducklake_from_agent_http and get_datatable_resource_from_agent_http, so in practice it usually means a datatable/ducklake resource fetch failed. Like the POST variant, the response body carrying the precise server-side reason is dropped.

Source

Thrown at backend/windmill-common/src/worker.rs:603

                url,
                response.status()
            )))
        }
    }

    pub async fn get<R: DeserializeOwned>(&self, url: &str) -> anyhow::Result<R> {
        let base_url = self.base_internal_url.clone();
        let response = self
            .client
            .get(format!("{}{}", base_url, url))
            .send()
            .await
            .map_err(|e| anyhow::anyhow!(e))?;
        let status = response.status();
        if status.is_success() {
            Ok(response.json().await?)
        } else {
            Err(anyhow::anyhow!(format!(
                "HTTP agent request GET {} failed {}",
                url,
                response.status()
            )))
        }
    }

    pub async fn get_bytes(&self, url: &str) -> anyhow::Result<Bytes> {
        let base_url = self.base_internal_url.clone();
        let response = self
            .client
            .get(format!("{}{}", base_url, url))
            .send()
            .await
            .map_err(|e| anyhow::anyhow!(e))?;
        if response.status().is_success() {
            Ok(response.bytes().await?)
        } else {

View on GitHub (pinned to e474e8803c)

Solutions

Check server logs for the corresponding GET request to see the discarded error body.
Verify the datatable/ducklake resource exists in the target workspace.
Check the worker's token permissions for the workspace/resource.
Retry on 5xx; fix the resource name/permissions for 4xx.

Example fix

// before
let resource = client.get::<DatatableResource>(&url).await?;
// after
let resource: DatatableResource = match client.get::<DatatableResource>(&url).await {
    Ok(r) => r,
    Err(e) if e.to_string().contains("503") || e.to_string().contains("500") => {
        tokio::time::sleep(Duration::from_secs(2)).await;
        client.get::<DatatableResource>(&url).await?
    }
    Err(e) => return Err(e),
};

Defensive patterns

Strategy: try-catch

Validate before calling

// Verify the resource exists before fetching
let exists: bool = client.get::<serde_json::Value>("/w/{ws}/datatables/list")
    .await
    .map(|v| v.to_string().contains(&table_name))
    .unwrap_or(false);
if !exists {
    return Err(anyhow::anyhow!("datatable '{table_name}' not found before GET"));
}

Try / catch

match client.get::<R>(&url).await {
    Ok(r) => r,
    Err(e) if e.to_string().contains("404") => {
        return Err(anyhow::anyhow!("resource missing, aborting (do not retry 404): {e}"));
    }
    Err(e) if e.to_string().contains("503") || e.to_string().contains("500") => {
        tokio::time::sleep(Duration::from_secs(2)).await;
        client.get::<R>(&url).await?
    }
    Err(e) => return Err(e),
}

Prevention

Check that referenced datatables/ducklake resources exist before dispatching jobs.
Test worker token read access to the workspace resource out-of-band.
Retry only on 5xx; treat 404/403 as configuration errors to fix.
Keep backend logs accessible to correlate the status code with the real error body.
Confirm the target instance has the datatable feature compiled/enabled.

When it happens

Trigger: Worker calls HttpClient::get for a datatable or ducklake resource (via the agent HTTP path) and the server replies 404 (resource/table not found), 403 (insufficient permissions), 4xx (bad request), or 5xx (server error).

Common situations: Referenced datatable/ducklake doesn't exist or was deleted; worker token can't read the resource in that workspace; server feature not compiled/enabled; wrong base_internal_url; transient 5xx while the backend restarts.

Related errors

AI-assisted analysis of windmill-labs/windmill@e474e8803c (2026-09-03). Data as JSON: /api/errors/1b0758000b8d3e98. Report an issue: GitHub.