databendlabs/databend · error

indexed key is not a GenericKV state entry at line

Error message

indexed key is not a GenericKV state entry at line {}: {}

What it means

After finding a key in key_to_state, classify_snapshot_orphan_root requires the corresponding state line to be a GenericKV entry, because orphan classification must decode the value payload (e.g. IndexMeta for __fd_index_by_id keys). If the indexed line's StateKind is anything else (e.g. System), the tool's index-to-kind invariant is violated and it bails naming the offending line number and display key.

Solutions

  1. Regenerate the snapshot dump with a matching metasrv version so these keys are exported as GenericKV state entries.
  2. Inspect the reported line in the dump file to see what record type the parser assigned it, and check the StateKind classification logic in filter_tenant's snapshot parser for that key prefix.
  3. Update the orphan-check prefix list (match in classify_snapshot_orphan_root) and the parser's StateKind classification together so a key with an orphan-checkable prefix is always parsed as GenericKV.
  4. Verify the dump file is not truncated or corrupted around the reported line.

Example fix

// before (parser misclassifies aux key)
StateKind::System { .. } if key.starts_with("__fd_table_lvt") => ...
// after
StateKind::GenericKV { value, .. } if key.starts_with("__fd_table_lvt") => ...
Defensive patterns

Strategy: type-guard

Validate before calling

fn ensure_generic_kv(line: &StateLine, key: &str) -> anyhow::Result<()> {
    anyhow::ensure!(
        matches!(line.kind, StateKind::GenericKV { .. }),
        "key {key} at line {} is not a GenericKV entry",
        line.line_no
    );
    Ok(())
}

Type guard

fn as_generic_kv(kind: &StateKind) -> Option<&Value> {
    if let StateKind::GenericKV { value, .. } = kind { Some(value) } else { None }
}

Try / catch

match as_generic_kv(&state_lines[idx].kind) {
    Some(value) => classify_orphan(key, value),
    None => {
        eprintln!("key {key} is not GenericKV; skipping orphan check");
        Ok(None)
    }
}

Prevention

When it happens

Trigger: filter_tenant::mark_all calls classify_snapshot_orphan_root on a key whose state_lines entry is not StateKind::GenericKV — i.e. the snapshot parser classified a key with an orphan-checkable prefix (__fd_table, __fd_index_by_id, __fd_marked_deleted_index, __fd_table_copied_file_lock, __fd_table_lvt, __fd_table_copied_files, __fd_table_tag) as some non-GenericKV kind.

Common situations: Snapshot dumps written by a newer/older metasrv that stores these aux keys with a different state-machine record type; hand-crafted or corrupted dump files; modifications to the snapshot line parser in filter_tenant that mis-classify these keys.

Understand the failure class

Background: "This is a bug, please report it": internal invariant violations, unreachable panics, and SNH errors explained — this error's family across 47 libraries.

Related errors

AI-assisted analysis of databendlabs/databend@288d84d76e (2026-09-11). Data as JSON: /api/errors/a80537d20d70943f. Report an issue: GitHub.

Appendix: source

Thrown at src/meta/process/src/filter_tenant.rs:347

                self.mark_required_key(&key, Decision::Drop, reason)?;
            }
        }

        self.drain_mark_queue()?;
        self.assert_every_state_line_marked()?;
        self.update_report();

        Ok(())
    }

    fn classify_snapshot_orphan_root(&self, key: &str) -> anyhow::Result<Option<String>> {
        let segments = split_key(key);

        let Some(index) = self.key_to_state.get(key).copied() else {
            anyhow::bail!("key is missing from state index: {key}");
        };
        let StateKind::GenericKV { value, .. } = &self.state_lines[index].kind else {
            anyhow::bail!(
                "indexed key is not a GenericKV state entry at line {}: {}",
                self.state_lines[index].line_no,
                self.state_lines[index].display_key()
            );
        };

        let reason = match segments.as_slice() {
            ["__fd_table", db_id, _table_name] => {
                let db_id = parse_u64_segment(key, db_id)?;
                if self.has_database_primary_record(db_id) {
                    return Ok(None);
                }
                Some(format!(
                    "snapshot orphan table name key with missing database id {db_id}"
                ))
            }
            ["__fd_index_by_id", _index_id] => {
                let index_meta = decode_as::<IndexMeta>(key, &value.data)?;

View on GitHub (pinned to 288d84d76e)