quickwit-oss/quickwit · error · StorageError

failed to URL-decode listed object key

Error message

failed to URL-decode listed object key `{encoded_key}`: {error}

What it means

When listing objects, S3 keys are URL-encoded by the service; Quickwit percent-decodes each key back into a path. If the encoded key is not valid percent-encoding or is not valid UTF-8 after decoding, this Internal error is raised. It means a listed object key cannot be interpreted as a storage path.

Solutions

1. Find and rename/re-upload the offending object in the bucket so its key is valid UTF-8 percent-encodable
2. List the bucket directly (aws s3 ls) around the reported prefix to identify the bad key
3. Avoid uploading objects with raw percent signs or non-UTF8 names into the Quickwit index bucket
4. If a backend mis-encodes keys, fix or upgrade that S3-compatible implementation

Example fix

// before (raw key with stray % uploaded out-of-band)
my-index/hotcache-100%.bin
// after
aws s3 mv s3://bucket/my-index/hotcache-100%.bin s3://bucket/my-index/hotcache-100pct.bin

// client-side pre-check before uploading out-of-band objects
if !key.is_ascii() || key.contains('%') { reject_or_rename(key); }

match storage.list(&prefix).try_collect::<Vec<_>>().await {
    Ok(objs) => use(objs),
    Err(e) if e.kind() == StorageErrorKind::Internal => inspect_bucket_keys_for_bad_encoding(),
    Err(e) => return Err(e.into()),
}

When it happens

Trigger: Calling `list` on an S3-compatible storage when a listed key contains invalid percent-escapes (e.g. a lone `%` not followed by two hex digits) or bytes that are not valid UTF-8 after decoding.

Common situations: Objects uploaded directly to the bucket by other tools with raw/non-UTF8 key names; a non-AWS S3 implementation that double-encodes or leaves keys unencoded inconsistently; manually renamed objects containing stray `%` characters.

Understand the failure class

Background: "Invalid URL" errors: why new URL(), URI.parse, and reqwest::Url reject your string — missing scheme, whitespace, and bad path format — this error's family across 39 libraries.

Related errors

• listed object ` ` is not under storage prefix
• listing objects is not supported for storage
• the returned multipart upload id was null
• `append_records` should be called with `position_opt: None`
• bundled file range overlaps split footer

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/3a71c1f207ca2997. Report an issue: GitHub.

Appendix: source

Thrown at quickwit/quickwit-storage/src/object_storage/s3_compatible_storage.rs:1224

            None => {
                warn!("listed object has no last modified time, skipping");
                continue;
            }
        };
        object_metadata.push(ObjectMetadata {
            path: relative_path,
            size: ByteSize(size_bytes),
            last_modified,
        });
    }
    Ok(object_metadata)
}

fn decode_list_object_key(encoded_key: &str) -> StorageResult<Cow<'_, str>> {
    percent_decode_str(encoded_key)
        .decode_utf8()
        .map_err(|error| {
            StorageErrorKind::Internal.with_error(anyhow::anyhow!(
                "failed to URL-decode listed object key `{encoded_key}`: {error}"
            ))
        })
}

/// Strips a storage prefix from an S3 key without interpreting it as a filesystem path.
fn strip_storage_prefix<'a>(key: &'a str, storage_prefix: &Path) -> Option<&'a str> {
    let prefix = storage_prefix.to_string_lossy();
    if prefix.is_empty() {
        return Some(key);
    }
    if key == prefix {
        return Some("");
    }
    if prefix.ends_with('/') {
        key.strip_prefix(prefix.as_ref())
    } else {
        let prefix_with_separator = format!("{prefix}/");

View on GitHub (pinned to a39730c5cd)