quickwit-oss/quickwit · error · StorageError

the returned multipart upload id was null

Error message

the returned multipart upload id was null

What it means

When starting an S3 multipart upload, the SDK's CreateMultipartUpload response must contain an `upload_id`. Quickwit unwraps it with `ok_or_else` and raises an Internal storage error if S3 returned null. This is an unexpected API response from the object store, since a successful create always carries an id.

Solutions

1. Check the object storage backend and any proxies in front of it for malformed CreateMultipartUpload responses (inspect with verbose/logging mode)
2. Upgrade or reconfigure the S3-compatible service (e.g. newer MinIO) to a version with correct XML responses
3. Retry the upload; if persistent, switch endpoint or use non-multipart (smaller) uploads to isolate the issue
4. Report the backend's raw response if it consistently omits upload_id

let upload = loop {
    match storage.put_multipart(&path).await {
        Ok(u) => break u,
        Err(e) if e.kind() == StorageErrorKind::Internal && attempts < 3 => { attempts += 1; continue; }
        Err(e) => return Err(e.into()),
    }
};

When it happens

Trigger: Calling `put_multipart` on S3CompatibleStorage when the CreateMultipartUpload response completes but has `upload_id: None` — typically caused by a proxy/MinIO/non-AWS S3-compatible service returning a malformed or non-standard response body.

Common situations: Using a buggy S3-compatible backend (MinIO, Ceph, vendor proxies) that strips or misparses the UploadResult XML; an intermediary (corporate proxy, some gateways) rewriting the response; SDK version parsing changes.

Understand the failure class

Background: "invalid response format", "malformed payload", "missing data field": when an API returns 200 but the response shape is wrong — this error's family across 23 libraries.

Related errors

• failed to URL-decode listed object key
• `append_records` should be called with `position_opt: None`
• bundled file range overlaps split footer
• bundled file range starts after it ends
• data too short (len= )

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/a510e4650d8e38b7. Report an issue: GitHub.

Appendix: source

Thrown at quickwit/quickwit-storage/src/object_storage/s3_compatible_storage.rs:478

        .map_err(|error| error.into_inner())?;
        Ok(())
    }

    async fn create_multipart_upload(&self, key: &str) -> StorageResult<MultipartUploadId> {
        let upload_id = aws_retry(&self.retry_params, || async {
            self.s3_client
                .create_multipart_upload()
                .bucket(self.bucket.clone())
                .set_checksum_algorithm(aws_checksum_algorithm(self.checksum_algorithm))
                .key(key)
                .send()
                .await
        })
        .await?
        .upload_id
        .ok_or_else(|| {
            StorageErrorKind::Internal
                .with_error(anyhow!("the returned multipart upload id was null"))
        })?;
        Ok(MultipartUploadId(upload_id))
    }

    /// Returns the MD5 of the byte range when the configured strategy is
    /// [`ChecksumAlgorithm::Md5`], otherwise `None` (no I/O performed).
    async fn maybe_compute_part_md5(
        &self,
        payload: &dyn crate::PutPayload,
        range: Range<u64>,
    ) -> io::Result<Option<md5::Digest>> {
        if !self.checksum_algorithm.is_md5() {
            return Ok(None);
        }
        let read = payload.range_byte_stream(range).await?.into_async_read();
        Ok(Some(compute_md5(read).await?))
    }

View on GitHub (pinned to a39730c5cd)