{"record":{"id":"fb2ffaad8883a0e5","repo":"windmill-labs/windmill","slug":"row-over-budget","errorCode":null,"errorMessage":"row over budget","messagePattern":"row over budget","errorType":"exception","errorClass":"std::io::Error (WriteZero)","httpStatus":null,"severity":"error","filePath":"backend/windmill-duckdb-ffi-internal/src/lib.rs","lineNumber":926,"sourceCode":"/// `None` is reported to the caller as \"too large\", which is only honest because\n/// the callers pass a `serde_json::Map` of `Value`s: serializing one cannot fail\n/// for any reason except the budget. A caller passing a type with a fallible\n/// `Serialize` would have its error silently retold as a size limit.\nfn to_raw_value_within<T: serde::Serialize>(value: &T, budget: usize) -> Option<Box<RawValue>> {\n    struct Budgeted {\n        buf: Vec<u8>,\n        left: usize,\n    }\n    impl std::io::Write for Budgeted {\n        fn write(&mut self, bytes: &[u8]) -> std::io::Result<usize> {\n            self.write_all(bytes)?;\n            Ok(bytes.len())\n        }\n        // `Vec<u8>` overrides this too: the default implementation loops over\n        // `write`, and serde_json emits a great many small pieces per row.\n        fn write_all(&mut self, bytes: &[u8]) -> std::io::Result<()> {\n            if bytes.len() > self.left {\n                return Err(std::io::Error::new(\n                    std::io::ErrorKind::WriteZero,\n                    \"row over budget\",\n                ));\n            }\n            self.left -= bytes.len();\n            self.buf.extend_from_slice(bytes);\n            Ok(())\n        }\n        fn flush(&mut self) -> std::io::Result<()> {\n            Ok(())\n        }\n    }\n\n    let mut writer = Budgeted { buf: Vec::new(), left: budget };\n    serde_json::to_writer(&mut writer, value).ok()?;\n    let json = String::from_utf8(writer.buf).ok()?;\n    // SAFETY: `to_writer` returned `Ok`, so `json` holds one complete, well-formed\n    // JSON value with no surrounding whitespace. Running out of budget is the only","sourceCodeStart":908,"sourceCodeEnd":944,"githubUrl":"https://github.com/windmill-labs/windmill/blob/e474e8803ce2ff5c2df09a58dab51d45f5c922ca/backend/windmill-duckdb-ffi-internal/src/lib.rs#L908-L944","documentation":"The DuckDB FFI writer serializes each JSON row into a bounded in-memory buffer. The custom `Write::write_all` checks the incoming bytes against the remaining budget (`self.left`) and raises WriteZero 'row over budget' when a single write would exceed the per-row size limit. This prevents unbounded memory use from oversized rows during JSON-to-DuckDB ingestion.","triggerScenarios":"Streaming a JSON result row via serde_json whose serialized size exceeds the row budget in write_all — i.e. one result row (wide columns, large strings, big nested JSON blobs) is too large for the configured row buffer.","commonSituations":"Scripts returning huge JSON objects per row (embedded base64 payloads, large arrays); selecting entire large JSON/text columns; flow steps aggregating many fields into a single row.","solutions":["Reduce the size of individual rows: drop or truncate oversized columns before returning results.","Split large payloads into multiple rows instead of one giant row.","Store large blobs in the Windmill S3 object store and keep only references/URLs in the row.","If this is a legitimate workload, increase the row budget in the duckdb FFI writer configuration."],"exampleFix":"// before\nSELECT payload FROM events; // payload is a 20MB JSON blob per row\n// after\nSELECT id, json_extract_string(payload, '$.summary') AS summary FROM events; // or store payload in S3 and reference it","handlingStrategy":"validation","validationCode":"// check serialized row size against budget before ingestion\nlet serialized = serde_json::to_vec(&row)?;\nconst ROW_BUDGET: usize = 4 * 1024 * 1024;\nif serialized.len() > ROW_BUDGET {\n    return Err(format!(\"row {} is {} bytes, exceeds {} budget; truncate or move payload to S3\", id, serialized.len(), ROW_BUDGET));\n}","typeGuard":null,"tryCatchPattern":"match write_all_result {\n    Err(e) if e.kind() == std::io::ErrorKind::WriteZero && e.to_string().contains(\"row over budget\") => {\n        eprintln!(\"row too large: drop/truncate large columns or store payload in S3\");\n    }\n    other => other?,\n}","preventionTips":["Never embed large blobs (base64, files) in individual result rows","Truncate or summarize oversized text/JSON columns in SQL before returning","Store large payloads in the S3 object store and reference them by key","Split wide denormalized rows into normalized child rows"],"tags":["duckdb","memory-limit","serialization","json"],"backgroundTag":"row-size-limit-exceeded","analyzedSha":"e474e8803ce2ff5c2df09a58dab51d45f5c922ca","analyzedAt":"2026-09-03T12:38:19.024Z","contentChangedAt":"2026-09-03T12:38:19.024Z","schemaVersion":2},"datasetVersion":"2026-09-08T10:18:20.063Z"}