quickwit-oss/quickwit · error

The prefix should have been prepended to the key before this

Error message

The prefix should have been prepended to the key before this method call.

What it means

`relative_path` strips the configured storage prefix from an object key via `strip_storage_prefix` and panics with this message if the key does not start with that prefix. The function's contract is that callers (e.g. `bulk_delete_multi`) have already applied the prefix when building keys, so a missing prefix is an internal invariant violation rather than a user-facing error. It converts S3 full keys back into storage-relative paths.

Source

Thrown at quickwit/quickwit-storage/src/object_storage/s3_compatible_storage.rs:391

    fn key(&self, relative_path: &Path) -> String {
        // FIXME: This may not work on Windows.
        let prefix = self.prefix.to_string_lossy();
        let relative_path = relative_path.to_string_lossy();
        if prefix.is_empty() {
            relative_path.to_string()
        } else if relative_path.is_empty() {
            prefix.to_string()
        } else if prefix.ends_with('/') {
            format!("{prefix}{relative_path}")
        } else {
            format!("{prefix}/{relative_path}")
        }
    }

    fn relative_path(&self, key: &str) -> PathBuf {
        // FIXME: This may not work on Windows.
        let relative_key = strip_storage_prefix(key, &self.prefix)
            .expect("The prefix should have been prepended to the key before this method call.");
        PathBuf::from(relative_key)
    }

    async fn put_single_part_single_try<'a>(
        &'a self,
        bucket: &'a str,
        key: &'a str,
        payload: Box<dyn crate::PutPayload>,
        len: u64,
    ) -> Result<(), Retry<StorageError>> {
        // For MD5 uploads, compute Content-MD5 before streaming the body.
        // The AWS SDK no-ops ChecksumAlgorithm::Md5, so MD5 must be sent via
        // the legacy Content-MD5 header (same as the multipart path does per part).
        let content_md5: Option<String> = self
            .maybe_compute_part_md5(payload.as_ref(), 0..len)
            .await
            .map_err(|err| Retry::Permanent(StorageError::from(err)))?
            .map(|digest| BASE64_STANDARD.encode(digest.0));

View on GitHub (pinned to a39730c5cd)

Solutions

1. Ensure all keys passed to delete/list operations are built by prepending the storage prefix (use the same key-building helper used by put operations).
2. If objects may live under other prefixes, filter keys by `key.starts_with(&self.prefix)` before calling `relative_path` instead of letting it panic.
3. Verify the storage `prefix` config matches the one used when the objects were written; a mismatched prefix causes both misses and this panic.
4. If you control the code, change `strip_storage_prefix` handling to return an error (e.g. skip or log) rather than expect, for externally influenced inputs.

Example fix

// before
let relative_key = strip_storage_prefix(key, &self.prefix)
    .expect("The prefix should have been prepended to the key before this method call.");
// after (caller-side guard)
if !key.starts_with(self.prefix.as_str()) { return PathBuf::from(key); }
let relative_key = strip_storage_prefix(key, &self.prefix)
    .expect("The prefix should have been prepended to the key before this method call.");

// Guard every key before any operation that funnels into relative_path
fn is_prefixed(key: &str, prefix: &str) -> bool { key.starts_with(prefix) }
// skip keys that do not belong to this storage instance
keys.retain(|k| is_prefixed(k, &storage_prefix));

fn strip_prefix_safe<'a>(key: &'a str, prefix: &'a str) -> Option<&'a str> {
    key.strip_prefix(prefix)
}

// This panics rather than returning an error; pre-filter instead of catching
let relative = match strip_storage_prefix(key, &prefix) {
    Some(k) => PathBuf::from(k),
    None => { tracing::warn!("skipping key with foreign prefix: {key}"); continue; }
};

When it happens

Trigger: A key passed to `relative_path` that does not begin with `self.prefix` — e.g. an object listed in the bucket that was created outside quickwit (manually uploaded or by an older deployment with a different prefix) and then targeted by a bulk delete, or a code path that composes keys without applying the prefix.

Common situations: Mixed-prefix buckets: the container/bucket holds objects under several prefixes (multiple quickwit indexes or manual uploads) and an operation iterates raw keys assuming the instance's prefix; also occurs after prefix configuration changes on an existing bucket, or on Windows paths due to the noted FIXME.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• input {} has kind {:?}, expected {:?}
• input {} has index_uid '{}', expected '{}'
• input {} has partition_id {}, expected {}
• input {} has sort_fields '{}', expected '{}'
• input {} has window {:?}, expected {:?}

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/5d7f453d49e256a4. Report an issue: GitHub.