quickwit-oss/quickwit · error

position of a Kafka partition should never be EOF

Error message

position of a Kafka partition should never be EOF

What it means

build_ingester_service wires the ingester service client. When the target node is the local node, the code expects the locally initialized ingester service to have been passed in via `ingester_opt`; `.expect` fires if it is None. This is an internal wiring invariant: on a self-node the service must already have been created during node startup.

Source

Thrown at quickwit/quickwit-indexing/src/source/kafka_source.rs:372

            self.state
                .assigned_partitions
                .insert(partition, partition_id.clone());

            let Some(current_position) = checkpoint.position_for_partition(&partition_id).cloned()
            else {
                continue;
            };
            let next_offset = match &current_position {
                Position::Beginning => Offset::Beginning,
                Position::Offset(offset) => {
                    let offset = offset
                        .as_i64()
                        .expect("Kafka offset should be stored as i64");
                    Offset::Offset(offset + 1)
                }
                Position::Eof(_) => {
                    panic!("position of a Kafka partition should never be EOF")
                }
            };
            self.state
                .current_positions
                .insert(partition, current_position);
            next_offsets.push((partition, next_offset));
        }
        info!(
            index_id=%self.source_runtime.index_id(),
            source_id=%self.source_runtime.source_id(),
            topic=%self.topic,
            group_id=%self.group_id,
            partitions=?partitions,
            "new partition assignment after rebalance",
        );
        assignment_tx
            .send(next_offsets)
            .context("Kafka consumer context was dropped")?;

View on GitHub (pinned to a39730c5cd)

Solutions

1. Add the ingester role to the node configuration so the ingester service is initialized at startup.
2. Fix the routing/placement logic so ingest requests for a node without an ingester go over gRPC to a remote ingester instead of the local shortcut.
3. Check node_config / peer seed setup: `is_self_node()` must only be true for nodes that actually run the ingester.
4. Verify cluster membership data (Chitchat) is not stale, which can make a remote node appear as self.

Example fix

// before
let ingester = ingester_opt.expect("ingester service should be initialized");
// after
let ingester = ingester_opt.ok_or_else(|| {
    anyhow::anyhow!("ingester service is not initialized on this node; is the ingester role enabled?")
})?;

// before bootstrapping ingest flows, verify the node runs the ingester role
let roles = &config.node_config.roles();
if !roles.contains(Role::Ingest) {
    return Err(anyhow!("node does not run the ingester; cannot use local ingester client"));
}

// callers of build_ingester_insert_change should map panics to startup failure:
let client = std::panic::catch_unwind(|| build_ingester_insert_change(...));

When it happens

Trigger: Calling build_ingester_insert_change / build_ingester_service for a self-node while the ingester service was never initialized (e.g. the node config does not enable the ingester role but some code path still asks for the local ingester client).

Common situations: Running a node whose config omits the ingester role while an index/ingest flow still routes to the local node; a code path or test that builds the service registry before the ingester actor is set up; misconfigured peer routing where a request thought to be local is resolved locally.

Understand the failure class

Background: "X is required", "must be set", "cannot be empty": the missing-required-config error family, from Vertex AI project/location to WeChat keys — this error's family across 18 libraries.

Related errors

• position of a Kinesis shard should never be EOF
• ingest API service with queues directory located at `{}` is
• node not found in pending
• OTP logs or traces do not support VRL transforms
• trying to replace in progress message

AI-assisted analysis of quickwit-oss/quickwit@a39730c5cd (2026-09-08). Data as JSON: /api/errors/045fa759ebd5465c. Report an issue: GitHub.