{"record":{"id":"38c484dad32909e3","repo":"windmill-labs/windmill","slug":"writer-close-task-panicked-e","errorCode":null,"errorMessage":"writer close task panicked: {e}","messagePattern":"writer close task panicked: (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"backend/windmill-object-store/src/lib.rs","lineNumber":1564,"sourceCode":"                Err(e) => tracing::error!(\"Error in blocking task: {:?}\", &e),\n            };\n        }\n        let close_result = task::spawn_blocking(move || {\n            writer.lock().unwrap().take().unwrap().close()?;\n            drop(writer);\n            Ok::<_, anyhow::Error>(())\n        })\n        .await;\n        match close_result {\n            Ok(Ok(())) => {}\n            Ok(Err(e)) => {\n                tracing::error!(\"Error closing S3 stream writer: {:?}\", e);\n                let _ = tx.send(Err(e)).await;\n            }\n            Err(e) => {\n                tracing::error!(\"S3 stream writer close task panicked: {:?}\", e);\n                let _ = tx\n                    .send(Err(anyhow::anyhow!(\"writer close task panicked: {e}\")))\n                    .await;\n            }\n        }\n        drop(ctx);\n        if let Err(e) = tokio::fs::remove_file(&path).await {\n            tracing::error!(\"Error removing temp file {}: {:?}\", path.display(), e);\n        }\n        Ok::<_, anyhow::Error>(())\n    });\n\n    Ok((\n        tokio_stream::wrappers::ReceiverStream::new(rx).boxed(),\n        ingest_stats,\n    ))\n}\n\n/// Decode the bytes of a Parquet file into a JSON array text (`[ {...}, {...} ]`)\n/// suitable for binding as a single SQL parameter and consuming with `OPENJSON`,","sourceCodeStart":1546,"sourceCodeEnd":1582,"githubUrl":"https://github.com/windmill-labs/windmill/blob/e474e8803ce2ff5c2df09a58dab51d45f5c922ca/backend/windmill-object-store/src/lib.rs#L1546-L1582","documentation":"When Windmill uploads a file to S3 it streams the body through a temp file with a spawned writer task. After the stream finishes, the code awaits that task's JoinHandle; if the close task panicked (JoinError), the panic is converted into this anyhow error and sent back over the result channel instead of propagating as a hard crash.","triggerScenarios":"The spawned S3 stream-writer task panics while finishing/closing the writer (internal bug in the async stream handling, a panicked assertion inside the writer future, or a task abort yielding JoinError). The temp file is then cleaned up and the error is surfaced to the caller awaiting the upload.","commonSituations":"Large or unusual file uploads through the S3 object store where the writer future hits an unexpected state; resource exhaustion (OOM killer interrupting the runtime) manifesting as aborted tasks; Rust runtime bugs or version mismatches in tokio/S3 crates.","solutions":["Check the logs just above this error for 'S3 stream writer close task panicked' to find the panic message and root cause","Retry the upload; transient aborts (e.g. worker shutdown mid-upload) resolve on retry","Inspect the temp-file directory permissions and disk space on the worker","Upgrade the Windmill backend and the object-store/S3 dependency crates to pick up panic fixes","Report a bug with the panic backtrace if it reproduces deterministically with a specific payload"],"exampleFix":"// before\nlet res = writer_handle.await.expect(\"writer task\");\n// after\nlet res = match writer_handle.await {\n    Ok(r) => r,\n    Err(e) => return Err(anyhow::anyhow!(\"writer close task panicked: {e}\")),\n};","handlingStrategy":"retry","validationCode":null,"typeGuard":null,"tryCatchPattern":"match upload_result {\n    Err(e) if e.to_string().contains(\"writer close task panicked\") => {\n        tracing::warn!(\"transient S3 writer panic, retrying: {e}\");\n        retry_upload(input, 2).await\n    }\n    Err(e) => Err(e),\n    Ok(v) => Ok(v),\n}","preventionTips":["Monitor worker logs for panics in the S3 writer path and upgrade on any reproducible one","Keep workers provisioned against OOM (memory limits, disk space on temp dirs)","Retry uploads idempotently — uploads to unique keys are safe to retry","Track Rust/tokio dependency versions and apply backend updates promptly"],"tags":["s3","async","panic","object-store"],"backgroundTag":"async-task-panic","analyzedSha":"e474e8803ce2ff5c2df09a58dab51d45f5c922ca","analyzedAt":"2026-09-03T12:38:19.024Z","contentChangedAt":"2026-09-03T12:38:19.024Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}