windmill-labs/windmill · error

writer close task panicked

Error message

writer close task panicked: {e}

What it means

When Windmill uploads a file to S3 it streams the body through a temp file with a spawned writer task. After the stream finishes, the code awaits that task's JoinHandle; if the close task panicked (JoinError), the panic is converted into this anyhow error and sent back over the result channel instead of propagating as a hard crash.

Solutions

  1. Check the logs just above this error for 'S3 stream writer close task panicked' to find the panic message and root cause
  2. Retry the upload; transient aborts (e.g. worker shutdown mid-upload) resolve on retry
  3. Inspect the temp-file directory permissions and disk space on the worker
  4. Upgrade the Windmill backend and the object-store/S3 dependency crates to pick up panic fixes
  5. Report a bug with the panic backtrace if it reproduces deterministically with a specific payload

Example fix

// before
let res = writer_handle.await.expect("writer task");
// after
let res = match writer_handle.await {
    Ok(r) => r,
    Err(e) => return Err(anyhow::anyhow!("writer close task panicked: {e}")),
};
Defensive patterns

Strategy: retry

Try / catch

match upload_result {
    Err(e) if e.to_string().contains("writer close task panicked") => {
        tracing::warn!("transient S3 writer panic, retrying: {e}");
        retry_upload(input, 2).await
    }
    Err(e) => Err(e),
    Ok(v) => Ok(v),
}

Prevention

When it happens

Trigger: The spawned S3 stream-writer task panics while finishing/closing the writer (internal bug in the async stream handling, a panicked assertion inside the writer future, or a task abort yielding JoinError). The temp file is then cleaned up and the error is surfaced to the caller awaiting the upload.

Common situations: Large or unusual file uploads through the S3 object store where the writer future hits an unexpected state; resource exhaustion (OOM killer interrupting the runtime) manifesting as aborted tasks; Rust runtime bugs or version mismatches in tokio/S3 crates.

Related errors

AI-assisted analysis of windmill-labs/windmill@e474e8803c (2026-09-03). Data as JSON: /api/errors/38c484dad32909e3. Report an issue: GitHub.

Appendix: source

Thrown at backend/windmill-object-store/src/lib.rs:1564

                Err(e) => tracing::error!("Error in blocking task: {:?}", &e),
            };
        }
        let close_result = task::spawn_blocking(move || {
            writer.lock().unwrap().take().unwrap().close()?;
            drop(writer);
            Ok::<_, anyhow::Error>(())
        })
        .await;
        match close_result {
            Ok(Ok(())) => {}
            Ok(Err(e)) => {
                tracing::error!("Error closing S3 stream writer: {:?}", e);
                let _ = tx.send(Err(e)).await;
            }
            Err(e) => {
                tracing::error!("S3 stream writer close task panicked: {:?}", e);
                let _ = tx
                    .send(Err(anyhow::anyhow!("writer close task panicked: {e}")))
                    .await;
            }
        }
        drop(ctx);
        if let Err(e) = tokio::fs::remove_file(&path).await {
            tracing::error!("Error removing temp file {}: {:?}", path.display(), e);
        }
        Ok::<_, anyhow::Error>(())
    });

    Ok((
        tokio_stream::wrappers::ReceiverStream::new(rx).boxed(),
        ingest_stats,
    ))
}

/// Decode the bytes of a Parquet file into a JSON array text (`[ {...}, {...} ]`)
/// suitable for binding as a single SQL parameter and consuming with `OPENJSON`,

View on GitHub (pinned to e474e8803c)