quickwit-oss/quickwit · error

Partition is owned by this indexing pipeline but is not at…

Error message

Partition is owned by this indexing pipeline but is not at the beginning. This should never happen! Please, report on https://github.com/quickwit-oss/quickwit/issues.

What it means

During partition acquisition, a partition that this indexing pipeline already owns must be resumable from its beginning (eof or beginning position). If an owned partition is found at a non-beginning position, the shared state invariant is broken, and acquire_partitions bails, asking for a bug report because this should be impossible by construction.

Solutions

1. File a bug at https://github.com/quickwit-oss/quickwit/issues with logs and the shard/partition state.
2. Reset the indexing pipeline state for the affected index/source so the partition positions are rebuilt from scratch.
3. Check for prior crashes or manual edits of persisted source state that could leave inconsistent positions.

match acquire_partitions(...) {
    Ok(shards) => shards,
    Err(e) => { log::error!("partition ownership invariant broken: {e}"); /* reset source state and resync shard positions */ }
}

When it happens

Trigger: acquire_partitions encounters a shard/partition whose publish token says it is owned by this pipeline, but its stored position is neither EOF nor at the beginning of the stream.

Common situations: Corrupted or stale shard position state (e.g., after an unclean restart or metastore/state divergence); a bug in position bookkeeping for Kinesis shards; mixing state from different pipeline generations.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

• Instance should be of type T.
• node not found in pending
• OTP logs or traces do not support VRL transforms
• position of a Kafka partition should never be EOF
• position of a Kinesis shard should never be EOF

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/430e859e24760aca. Report an issue: GitHub.

Appendix: source

Thrown at quickwit/quickwit-indexing/src/source/queue_sources/shared_state.rs:157

        let mut shards = Vec::new();
        let mut re_acquired_shards = Vec::new();
        for sub in open_shard_resp.subresponses {
            // we could also just cast the shard_id back to a partition_id
            let partition_id = partitions[sub.subrequest_id as usize].clone();
            let shard = sub.open_shard();
            let position = shard.publish_position_inclusive.clone().unwrap_or_default();
            let is_owned = sub.open_shard().publish_token.as_deref() == Some(publish_token);
            let update_datetime = OffsetDateTime::from_unix_timestamp(shard.update_timestamp)
                .context("Invalid shard update timestamp")?;
            let is_stale =
                OffsetDateTime::now_utc() - update_datetime > self.reacquire_grace_period;
            if position.is_eof() || (is_owned && position.is_beginning()) {
                shards.push((partition_id, position));
            } else if !is_owned && is_stale {
                info!(previous_token = shard.publish_token, "shard re-acquired");
                re_acquired_shards.push(shard.shard_id().clone());
            } else if is_owned && !position.is_beginning() {
                bail!(
                    "Partition is owned by this indexing pipeline but is not at the beginning. This should never happen! Please, report on https://github.com/quickwit-oss/quickwit/issues."
                )
            }
        }

        if re_acquired_shards.is_empty() {
            return Ok(shards);
        }

        // re-acquire shards that have a token that is not the local token
        let acquire_shard_resp = self
            .metastore
            .acquire_shards(AcquireShardsRequest {
                index_uid: Some(self.source_uid.index_uid.clone()),
                source_id: self.source_uid.source_id.clone(),
                shard_ids: re_acquired_shards,
                publish_token: publish_token.to_string(),
            })

View on GitHub (pinned to a39730c5cd)