{"id":"c40ced9a425237f4","repo":"BurntSushi/ripgrep","slug":"configured-allocation-limit-exceeded","errorCode":null,"errorMessage":"configured allocation limit ({}) exceeded","messagePattern":"configured allocation limit \\((.+?)\\) exceeded","errorType":"console","errorClass":null,"httpStatus":null,"severity":"error","filePath":"crates/searcher/src/line_buffer.rs","lineNumber":39,"sourceCode":"    /// This is the default.\n    Eager,\n    /// Limit the amount of additional memory allocated to the given size. If\n    /// a line is found that requires more memory than is allowed here, then\n    /// stop reading and return an error.\n    Error(usize),\n}\n\nimpl Default for BufferAllocation {\n    fn default() -> BufferAllocation {\n        BufferAllocation::Eager\n    }\n}\n\n/// Create a new error to be used when a configured allocation limit has been\n/// reached.\npub(crate) fn alloc_error(limit: usize) -> io::Error {\n    let msg = format!(\"configured allocation limit ({}) exceeded\", limit);\n    io::Error::new(io::ErrorKind::Other, msg)\n}\n\n/// The behavior of binary detection in the line buffer.\n///\n/// Binary detection is the process of _heuristically_ identifying whether a\n/// given chunk of data is binary or not, and then taking an action based on\n/// the result of that heuristic. The motivation behind detecting binary data\n/// is that binary data often indicates data that is undesirable to search\n/// using textual patterns. Of course, there are many cases in which this isn't\n/// true, which is why binary detection is disabled by default.\n#[derive(Clone, Copy, Debug, Eq, PartialEq)]\npub(crate) enum BinaryDetection {\n    /// No binary detection is performed. Data reported by the line buffer may\n    /// contain arbitrary bytes.\n    None,\n    /// The given byte is searched in all contents read by the line buffer. If\n    /// it occurs, then the data is considered binary and the line buffer acts\n    /// as if it reached EOF. The line buffer guarantees that this byte will","sourceCodeStart":21,"sourceCodeEnd":57,"githubUrl":"https://github.com/BurntSushi/ripgrep/blob/3fce3b5bb0236da2df6d99672afb8a719642eca7/crates/searcher/src/line_buffer.rs#L21-L57","documentation":"This is an io::Error produced by grep-searcher's LineBuffer (the streaming line reader that backs the Searcher). It is returned from LineBufferReader::fill / LineBuffer::fill via ensure_capacity (crates/searcher/src/line_buffer.rs:499) only when the buffer was built with BufferAllocation::Error(limit) instead of the default BufferAllocation::Eager. The value printed in the message is capacity+limit (line 512), i.e. the total heap budget the searcher is permitted to use; once a single logical line needs more room than that budget allows, the buffer refuses to grow and surfaces this error rather than OOM-ing the process. The public API that turns it on is SearcherBuilder::heap_limit (crates/searcher/src/searcher/mod.rs:457), which maps the user-supplied byte count into capacity + BufferAllocation::Error(additional) at searcher/mod.rs:224-233.","triggerScenarios":"Calling SearcherBuilder::new().heap_limit(Some(N)).build() and then searching a file whose longest line (including any before/after context window, since the buffer must hold the whole window) exceeds the configured N bytes. Concretely: ensure_capacity hits BufferAllocation::Error(limit), computes used = buf.len() - capacity, and when min(len*2, limit - used) == 0 it returns Err(alloc_error(capacity+limit)) (line_buffer.rs:508-513). It is also returned up front when heap_limit is set so low that no buffer can be allocated at all (searcher/mod.rs:996 and :1021, when additional == 0). Streaming search (Searcher::search_reader) with a fixed heap_limit over a file with a very long single line is the canonical trigger; multi-line mode with context enabled makes it more likely because the buffer must hold the context window plus the matched line.","commonSituations":"Operators running ripgrep-like tools with a memory cap (e.g. a --max-columns or memory-limit flag, or an embedded grep-searcher inside a constrained service) on minified JS, concatenated CSV/TSV without newlines, log files with one giant line, SQL dumps, or base64 blobs. Setting heap_limit to a small value while also enabling before_context/after_context (which expand the required buffer) is a frequent misconfiguration. Migrating from the default (no limit, Eager allocation) to a capped heap_limit without raising it for files known to have long lines is the typical regression. Enabling memory-map search disabled (MmapChoice::never) plus heap_limit on a huge-line file guarantees the error because there is no fallback strategy.","solutions":["If you can afford the memory, raise or remove heap_limit: pass None (or a larger N) to SearcherBuilder::heap_limit. The default Eager strategy will grow the buffer up to available memory and never produce this error.","If the offending file genuinely has pathologically long lines, pre-process or skip it (e.g. enable BinaryDetection::Quit, filter by file size, or use --no-line-buffer style streaming) so the buffer never has to hold the whole line.","Reduce the buffer pressure by lowering before_context/after_context (Config::before_context / after_context) so the required buffer window is smaller relative to your heap_limit.","Enable memory-map search (MmapChoice::auto or ::when appropriate) via SearcherBuilder::memory_map; mmap reads do not consume the line buffer's heap budget, sidestepping the limit entirely.","If the cap must stay tiny, set heap_limit to at least a few times the size of your longest expected line plus context; do not set it to 0 or to a value <= the longest line length.","Handle the error at the call site: match on io::ErrorKind::Other and the 'configured allocation limit' message, then skip the file or fall back to a streaming/partial search rather than aborting the whole search."],"exampleFix":"// before\nuse grep_searcher::{Searcher, SearcherBuilder};\nlet mut searcher = SearcherBuilder::new()\n    .heap_limit(Some(64 * 1024)) // 64 KiB cap\n    .build();\nlet result = searcher.search_reader(&matcher, &mut file, &mut sink);\n// -> io error: \"configured allocation limit (65536) exceeded\" on long-line files\n\n// after — raise the cap to fit the worst-case line + context, or remove it\nuse grep_searcher::{Searcher, SearcherBuilder};\nlet mut searcher = SearcherBuilder::new()\n    .heap_limit(Some(64 * 1024 * 1024)) // 64 MiB, well above any single line\n    .build();\n// or, if memory is not a concern, omit heap_limit entirely to use Eager:\n// let mut searcher = SearcherBuilder::new().build();","handlingStrategy":"validation","validationCode":"// Before searching, size heap_limit against the file's longest line + context window.\nuse std::fs::File;\nuse std::io::{self, Read, BufReader};\n\n/// Returns the length (in bytes) of the longest line in `path`, or an io::Error.\n/// Use this to pick a heap_limit that is guaranteed not to trip the alloc_error.\nfn longest_line_len(path: &std::path::Path) -> io::Result<usize> {\n    let f = File::open(path)?;\n    let mut r = BufReader::new(f);\n    let mut buf = [0u8; 64 * 1024];\n    let mut max = 0usize;\n    let mut cur = 0usize;\n    loop {\n        let n = r.read(&mut buf)?;\n        if n == 0 { break; }\n        for &b in &buf[..n] {\n            if b == b'\\n' {\n                if cur > max { max = cur; }\n                cur = 0;\n            } else {\n                cur += 1;\n            }\n        }\n    }\n    if cur > max { max = cur; }\n    Ok(max)\n}\n\n// Then choose heap_limit so that capacity (64 KiB) + additional > longest_line + context.\n// Keep a generous multiplier; the line buffer must hold the line AND one more byte to detect EOF.\nlet longest = longest_line_len(&path)?;\nlet context = searcher_before_ctx + searcher_after_ctx;\nlet needed = longest.saturating_add(context).saturating_add(1);\nlet heap_limit = if needed < 64 * 1024 { 64 * 1024 } else { needed };\nlet mut searcher = SearcherBuilder::new().heap_limit(Some(heap_limit)).build();","typeGuard":"// Rust: there is no dedicated error type; narrow by kind + message substring.\nfn is_allocation_limit_error(e: &std::io::Error) -> bool {\n    use std::io::ErrorKind;\n    e.kind() == ErrorKind::Other\n        && e.to_string().contains(\"configured allocation limit\")\n}\n\n// Usage in a sink/search call:\nmatch searcher.search_reader(&matcher, &mut file, &mut sink) {\n    Ok(()) => {},\n    Err(ref e) if is_allocation_limit_error(e) => {\n        // line too long for the configured heap_limit; handle gracefully\n    }\n    Err(other) => return Err(other.into()),\n}","tryCatchPattern":"// Recommended pattern when iterating many files with a heap cap:\nfor path in paths {\n    let result = searcher.search_path(&matcher, path, &mut sink);\n    match result {\n        Ok(()) => {},\n        Err(ref e) if is_allocation_limit_error(&e.into_io().unwrap_or_else(|_| io::Error::new(io::ErrorKind::Other, e.to_string()))) => {\n            // Log and skip; do NOT abort the whole search over one oversized line.\n            tracing::warn!(?path, \"skipped: line exceeds heap_limit\");\n            continue;\n        }\n        Err(e) => return Err(e),\n    }\n}","preventionTips":["Do not set SearcherBuilder::heap_limit to a value smaller than the longest single line you expect to search; the line buffer must hold the entire line plus one byte to detect EOF.","Remember context: when before_context/after_context are non-zero, the buffer must hold the line AND the context window, so budget heap_limit accordingly.","Prefer MmapChoice when searching huge files with long lines — mmap does not consume the heap budget.","For embedded use in servers/CLIs, expose heap_limit as a tunable (env var or flag) and document that lowering it trades memory safety for this error on long-line inputs.","If you only need line-oriented search and never multi-line/context, you can stream with a tiny heap_limit by also disabling context (before_context = after_context = 0) and accepting that files with a single huge line will be skipped.","Validate input shape before searching: pre-scan for line length on untrusted inputs (logs, minified assets, dumps) so you can size heap_limit or route the file to an mmap/streaming path instead of discovering the limit at search time."],"tags":["rust","grep-searcher","memory","io-error","configuration","ripgrep"],"analyzedSha":"3fce3b5bb0236da2df6d99672afb8a719642eca7","analyzedAt":"2026-08-06T01:39:05.073Z","schemaVersion":2}