{"record":{"id":"18b2c52546ba9574","repo":"influxdata/influxdb","slug":"if-we-can-remove-a-value-from-the-interned-strings","errorCode":null,"errorMessage":"If we can remove a value from the interned strings, we must also be able to remove a value from the packed strings.","messagePattern":"If we can remove a value from the interned strings, we must also be able to remove a value from the packed strings\\.","errorType":"panic","errorClass":null,"httpStatus":null,"severity":"warning","filePath":"core/table_batch/src/builder/column_writer/dictionary.rs","lineNumber":80,"sourceCode":"    ///\n    /// If this call removes the sole reference to a value in this buffer, the value will not appear\n    /// in the decoded result, and is removed from the buffer containing all previously-pushed\n    /// strings. This allows us to avoid storing and transmitting unnecessary data, at the cost of\n    /// a slightly more expensive operation when we have to remove a value due to recovering from a\n    /// partial write.\n    ///\n    /// # Panics\n    ///\n    /// Panics if no values remain in `self`.\n    pub(crate) fn drop_last_value(&mut self) {\n        let last_pushed = self.encoded.pop()\n            .expect(\"If you call `drop_last_value`, the tag buffer must contain at least one value, but it contained zero\");\n\n        // And we should remove it from the packed strings as well if it failed, just to get more\n        // predictable behavior and avoid sending some data if possible.\n        if !self.encoded.contains(&last_pushed) {\n            let removed = self.keys.pop()\n                .expect(\"If we can remove a value from the interned strings, we must also be able to remove a value from the packed strings.\");\n\n            self.id_map.remove(&removed);\n        }\n    }\n\n    pub(crate) fn finish(self) -> Option<InternedStrings> {\n        let dictionary = self.keys.finish()?;\n\n        // Invariant: the key -> ID map and the buffer containing the encoded\n        // keys must always agree on the number of unique keys observed.\n        debug_assert_eq!(\n            self.id_map.len(),\n            dictionary.offsets.len() - 1, // 0 starting offset\n        );\n\n        if self.encoded.is_empty() {\n            None\n        } else {","sourceCodeStart":62,"sourceCodeEnd":98,"githubUrl":"https://github.com/influxdata/influxdb/blob/d28e26e048401c53cbb98cf2d6ab0cf1e98048ca/core/table_batch/src/builder/column_writer/dictionary.rs#L62-L98","documentation":"After dropping the last encoded value, DictionaryBuffer checks whether that dictionary ID is still referenced; if not, it removes the interned string from the packed-strings buffer via keys.pop(). The .expect encodes the invariant that keys (StringBuffer of unique dictionary entries) and encoded (the ID stream) stay in sync: whenever encoded held an ID, keys must hold the corresponding string. It can only fire if push_str/drop_last_value bookkeeping between id_map, keys, and encoded was already corrupted.","triggerScenarios":"No single call produces it directly; it follows a prior state corruption such as removing encoded IDs without adjusting keys, pushing a duplicate dictionary entry so keys and id_map disagree, or reordering drop calls across buffers. In correct code the invariant debug_assert_eq! in finish() also asserts the same agreement, so this expect is a mid-operation version of that check.","commonSituations":"Seen only when someone modifies DictionaryBuffer/push_str/drop_last_value logic (e.g. adding dedupe or compaction) and breaks the id_map <-> keys <-> encoded relationship; or a data race mutating the builder from two threads (the builder is not Sync).","solutions":["Reproduce with a push/drop sequence property test (the module already uses proptest) to find the minimal sequence that desynchronizes the buffers.","Review any recent changes to push_str or drop_last_value for missed id_map/keys updates — the three fields must be updated together.","If multi-threaded writes share a Builder, stop: builders are single-threaded per column; concurrent mutation can corrupt the packed strings.","Add the failing sequence as a regression test asserting debug_assert_eq!(id_map.len(), dictionary.offsets.len() - 1) in finish()."],"exampleFix":null,"handlingStrategy":"validation","validationCode":"// Keep the three fields in sync: after any mutation, re-check the invariant.\n// (Defensive check inside the crate, near the mutation site.)\nassert_eq!(\n    self.id_map.len(),\n    self.keys.len(),\n    \"id_map and packed strings desynchronized\"\n);","typeGuard":null,"tryCatchPattern":null,"preventionTips":["Treat id_map, keys, and encoded as one unit — update all three in the same code path (push_str/drop_last_value).","Never mutate the builder from multiple threads; builders are single-threaded per column.","Run the module's proptest push/drop suite after touching dictionary encoding code."],"tags":["rust","influxdb","internal-invariant","dictionary-encoding","panic"],"backgroundTag":"internal-invariant-violation","analyzedSha":"d28e26e048401c53cbb98cf2d6ab0cf1e98048ca","analyzedAt":"2026-08-16T19:53:34.623Z","schemaVersion":2},"datasetVersion":"2026-08-16T23:17:17.608Z"}