windmill-labs/windmill · error

writer close task panicked: {e}

Error message

writer close task panicked: {e}

What it means

When Windmill uploads a file to S3 it streams the body through a temp file with a spawned writer task. After the stream finishes, the code awaits that task's JoinHandle; if the close task panicked (JoinError), the panic is converted into this anyhow error and sent back over the result channel instead of propagating as a hard crash.

Source

Thrown at backend/windmill-object-store/src/lib.rs:1564

                Err(e) => tracing::error!("Error in blocking task: {:?}", &e),
            };
        }
        let close_result = task::spawn_blocking(move || {
            writer.lock().unwrap().take().unwrap().close()?;
            drop(writer);
            Ok::<_, anyhow::Error>(())
        })
        .await;
        match close_result {
            Ok(Ok(())) => {}
            Ok(Err(e)) => {
                tracing::error!("Error closing S3 stream writer: {:?}", e);
                let _ = tx.send(Err(e)).await;
            }
            Err(e) => {
                tracing::error!("S3 stream writer close task panicked: {:?}", e);
                let _ = tx
                    .send(Err(anyhow::anyhow!("writer close task panicked: {e}")))
                    .await;
            }
        }
        drop(ctx);
        if let Err(e) = tokio::fs::remove_file(&path).await {
            tracing::error!("Error removing temp file {}: {:?}", path.display(), e);
        }
        Ok::<_, anyhow::Error>(())
    });

    Ok((
        tokio_stream::wrappers::ReceiverStream::new(rx).boxed(),
        ingest_stats,
    ))
}

/// Decode the bytes of a Parquet file into a JSON array text (`[ {...}, {...} ]`)
/// suitable for binding as a single SQL parameter and consuming with `OPENJSON`,

View on GitHub (pinned to e474e8803c)

Solutions

1. Check the logs just above this error for 'S3 stream writer close task panicked' to find the panic message and root cause
2. Retry the upload; transient aborts (e.g. worker shutdown mid-upload) resolve on retry
3. Inspect the temp-file directory permissions and disk space on the worker
4. Upgrade the Windmill backend and the object-store/S3 dependency crates to pick up panic fixes
5. Report a bug with the panic backtrace if it reproduces deterministically with a specific payload

Example fix

// before
let res = writer_handle.await.expect("writer task");
// after
let res = match writer_handle.await {
    Ok(r) => r,
    Err(e) => return Err(anyhow::anyhow!("writer close task panicked: {e}")),
};

match upload_result {
    Err(e) if e.to_string().contains("writer close task panicked") => {
        tracing::warn!("transient S3 writer panic, retrying: {e}");
        retry_upload(input, 2).await
    }
    Err(e) => Err(e),
    Ok(v) => Ok(v),
}

When it happens

Trigger: The spawned S3 stream-writer task panics while finishing/closing the writer (internal bug in the async stream handling, a panicked assertion inside the writer future, or a task abort yielding JoinError). The temp file is then cleaned up and the error is surfaced to the caller awaiting the upload.

Common situations: Large or unusual file uploads through the S3 object store where the writer future hits an unexpected state; resource exhaustion (OOM killer interrupting the runtime) manifesting as aborted tasks; Rust runtime bugs or version mismatches in tokio/S3 crates.

Related errors

• Parquet S3 input requires the `parquet` feature to be enable
• CSV S3 input requires the `parquet` feature to be enabled on
• errorMsg ?? 'Job failed'
• Job was cancelled
• ${timeoutMessage}

AI-assisted analysis of windmill-labs/windmill@e474e8803c (2026-09-03). Data as JSON: /api/errors/38c484dad32909e3. Report an issue: GitHub.