influxdata/influxdb · warning

If we can remove a value from the interned strings, we must

Error message

If we can remove a value from the interned strings, we must also be able to remove a value from the packed strings.

What it means

After dropping the last encoded value, DictionaryBuffer checks whether that dictionary ID is still referenced; if not, it removes the interned string from the packed-strings buffer via keys.pop(). The .expect encodes the invariant that keys (StringBuffer of unique dictionary entries) and encoded (the ID stream) stay in sync: whenever encoded held an ID, keys must hold the corresponding string. It can only fire if push_str/drop_last_value bookkeeping between id_map, keys, and encoded was already corrupted.

Source

Thrown at core/table_batch/src/builder/column_writer/dictionary.rs:80

    ///
    /// If this call removes the sole reference to a value in this buffer, the value will not appear
    /// in the decoded result, and is removed from the buffer containing all previously-pushed
    /// strings. This allows us to avoid storing and transmitting unnecessary data, at the cost of
    /// a slightly more expensive operation when we have to remove a value due to recovering from a
    /// partial write.
    ///
    /// # Panics
    ///
    /// Panics if no values remain in `self`.
    pub(crate) fn drop_last_value(&mut self) {
        let last_pushed = self.encoded.pop()
            .expect("If you call `drop_last_value`, the tag buffer must contain at least one value, but it contained zero");

        // And we should remove it from the packed strings as well if it failed, just to get more
        // predictable behavior and avoid sending some data if possible.
        if !self.encoded.contains(&last_pushed) {
            let removed = self.keys.pop()
                .expect("If we can remove a value from the interned strings, we must also be able to remove a value from the packed strings.");

            self.id_map.remove(&removed);
        }
    }

    pub(crate) fn finish(self) -> Option<InternedStrings> {
        let dictionary = self.keys.finish()?;

        // Invariant: the key -> ID map and the buffer containing the encoded
        // keys must always agree on the number of unique keys observed.
        debug_assert_eq!(
            self.id_map.len(),
            dictionary.offsets.len() - 1, // 0 starting offset
        );

        if self.encoded.is_empty() {
            None
        } else {

View on GitHub (pinned to d28e26e048)

Solutions

1. Reproduce with a push/drop sequence property test (the module already uses proptest) to find the minimal sequence that desynchronizes the buffers.
2. Review any recent changes to push_str or drop_last_value for missed id_map/keys updates — the three fields must be updated together.
3. If multi-threaded writes share a Builder, stop: builders are single-threaded per column; concurrent mutation can corrupt the packed strings.
4. Add the failing sequence as a regression test asserting debug_assert_eq!(id_map.len(), dictionary.offsets.len() - 1) in finish().

// Keep the three fields in sync: after any mutation, re-check the invariant.
// (Defensive check inside the crate, near the mutation site.)
assert_eq!(
    self.id_map.len(),
    self.keys.len(),
    "id_map and packed strings desynchronized"
);

When it happens

Trigger: No single call produces it directly; it follows a prior state corruption such as removing encoded IDs without adjusting keys, pushing a duplicate dictionary entry so keys and id_map disagree, or reordering drop calls across buffers. In correct code the invariant debug_assert_eq! in finish() also asserts the same agreement, so this expect is a mid-operation version of that check.

Common situations: Seen only when someone modifies DictionaryBuffer/push_str/drop_last_value logic (e.g. adding dedupe or compaction) and breaks the id_map <-> keys <-> encoded relationship; or a data race mutating the builder from two threads (the builder is not Sync).

Related errors

• must have offset
• num_columns_in_parallel should be above zero
• sharder mapped input to non-existant bucket
• mapped to out-of-bounds shard
• If you call `drop_last_value`, the tag buffer must contain a

AI-assisted analysis of influxdata/influxdb@d28e26e048 (2026-08-16). Data as JSON: /api/errors/18b2c52546ba9574. Report an issue: GitHub.