quickwit-oss/quickwit · error

position of a Kinesis shard should never be EOF

Error message

position of a Kinesis shard should never be EOF

What it means

build_indexing_service wires the indexing service client. For the local node it expects `indexing_service_opt` to hold the initialized indexing service mailbox; `.expect` fires if it is None. The indexing service must be created during node startup before any client (including the local, network-bypassing one) is built.

Source

Thrown at quickwit/quickwit-indexing/src/source/kinesis/kinesis_source.rs:156

    ) {
        if self.state.shard_consumers.contains_key(&shard_id) {
            info!(
                stream_name = %self.stream_name,
                shard_id = %shard_id,
                "Shard consumer already exists, skipping creation."
            );
            return;
        }

        let partition_id = PartitionId::from(shard_id.as_str());
        let from_position = checkpoint
            .position_for_partition(&partition_id)
            .cloned()
            .unwrap_or(Position::Beginning);
        let from_sequence_number_exclusive = match &from_position {
            Position::Beginning => None,
            Position::Offset(offset) => Some(offset.to_string()),
            Position::Eof(_) => panic!("position of a Kinesis shard should never be EOF"),
        };
        info!(
            stream_name = %self.stream_name,
            shard_id = %shard_id,
            start_position = ?from_position,
            "Spawning new shard consumer"
        );
        let shard_consumer = ShardConsumer::new(
            self.stream_name.clone(),
            shard_id.clone(),
            from_sequence_number_exclusive,
            self.backfill_mode_enabled,
            self.kinesis_client.clone(),
            self.shard_consumers_tx.clone(),
            self.retry_params,
        );
        let _shard_consumer_handle = shard_consumer.spawn(ctx);
        let shard_consumer_state = ShardConsumerState {

View on GitHub (pinned to a39730c5cd)

Solutions

1. Enable the indexer role in the node configuration so the indexing service is initialized at startup.
2. Fix control-plane/placement logic so indexing tasks are only assigned to nodes that run the indexer.
3. Ensure node initialization order creates the indexing service before build_indexer_insert_change runs.
4. Check Chitchat membership for stale self-node detection.

Example fix

// before
let indexing_service_mailbox =
    indexing_service_opt.expect("indexing service should be initialized");
// after
let indexing_service_mailbox = indexing_service_opt.ok_or_else(|| {
    anyhow::anyhow!("indexing service is not initialized on this node; is the indexer role enabled?")
})?;

// ensure the indexer role is enabled before building indexing clients
if !config.indexer.is_configured() {
    return Err(anyhow!("indexer role missing from node config; indexing service will not be initialized"));
}

// isolate service construction during node bootstrap:
let res = std::panic::catch_unwind(|| build_indexer_insert_change(deps));

When it happens

Trigger: Building an indexing client for a self-node when the indexing service was never started — e.g. the node config disables the indexer role but indexing tasks are still scheduled onto this node.

Common situations: A node launched without the indexer/indexer role receiving indexing assignments; control-plane placement scheduling tasks to a node that does not run the indexer; tests or code constructing the service map before the indexing actors exist.

Understand the failure class

Background: "X is required", "must be set", "cannot be empty": the missing-required-config error family, from Vertex AI project/location to WeChat keys — this error's family across 18 libraries.

Related errors

• position of a Kafka partition should never be EOF
• node not found in pending
• OTP logs or traces do not support VRL transforms
• trying to replace in progress message
• Partition is owned by this indexing pipeline but is not at t

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/29a3bf3032d57213. Report an issue: GitHub.