{"record":{"id":"d4341cbf3f344263","repo":"risingwavelabs/risingwave","slug":"turbopuffer-string-document-id-exceeds-64-bytes","errorCode":null,"errorMessage":"Turbopuffer string document id exceeds 64 bytes","messagePattern":"Turbopuffer string document id exceeds 64 bytes","errorType":"http","errorClass":"SinkError::Http","httpStatus":null,"severity":"error","filePath":"src/connector/src/sink/turbopuffer.rs","lineNumber":528,"sourceCode":"                Ok(format!(\"{}/v2/namespaces/{}\", self.base_url, namespace))\n            }\n        }\n    }\n\n    // Turbopuffer document IDs are unsigned 64-bit integers, UUIDs, or strings up to 64 bytes.\n    // RisingWave UUID IDs can be represented with varchar.\n    fn id_for_row(&self, row: &impl Row) -> Result<DocumentId> {\n        let datum = row.datum_at(self.pk_index).ok_or_else(|| {\n            SinkError::Http(anyhow!(\"Turbopuffer document id column cannot be null\"))\n        })?;\n        match datum {\n            ScalarRefImpl::Int16(value) => Ok(document_id_from_i64(value as i64)),\n            ScalarRefImpl::Int32(value) => Ok(document_id_from_i64(value as i64)),\n            ScalarRefImpl::Int64(value) => Ok(document_id_from_i64(value)),\n            ScalarRefImpl::Serial(value) => Ok(document_id_from_i64(value.into_inner())),\n            ScalarRefImpl::Utf8(value) => {\n                if value.len() > 64 {\n                    return Err(SinkError::Http(anyhow!(\n                        \"Turbopuffer string document id exceeds 64 bytes\"\n                    )));\n                }\n                Ok(DocumentId::String(value.to_owned()))\n            }\n            _ => Err(SinkError::Http(anyhow!(\n                \"Turbopuffer document id column must be an integer or varchar\"\n            ))),\n        }\n    }\n\n    fn upsert_row(&self, row: &impl Row, id: DocumentId) -> Result<Map<String, Value>> {\n        let mut value = self.row_encoder.encode(row)?;\n        value.insert(\n            \"id\".to_owned(),\n            serde_json::to_value(id).expect(\"serialize document id\"),\n        );\n        Ok(value)","sourceCodeStart":510,"sourceCodeEnd":546,"githubUrl":"https://github.com/risingwavelabs/risingwave/blob/6469eb736d691e8e9b8a419a57edd6429ca77417/src/connector/src/sink/turbopuffer.rs#L510-L546","documentation":"Raised in id_for_row when the primary-key value used as Turbopuffer document id is a string longer than 64 bytes. Turbopuffer accepts string document IDs of at most 64 bytes, so rows whose PK string exceeds this limit cannot be written and the write fails at runtime while building the document id.","triggerScenarios":"Writing a row whose varchar pk value exceeds 64 bytes (e.g. long concatenations, hashes-as-hex, full URLs used as keys).","commonSituations":"Using long natural keys (composite keys joined with delimiters, file paths, URLs) as the sink pk; switching from numeric keys to string keys without truncation policy.","solutions":["Shorten the pk upstream, e.g. hash it: `md5(key)`, `sha256(key)` truncated, or use a numeric surrogate key.","Use an Int64/Serial pk instead of a long varchar key.","Truncate to <=64 bytes only if uniqueness is preserved (not recommended blindly).","Validate key lengths at ingestion time before the sink sees them."],"exampleFix":"// before\nCREATE MATERIALIZED VIEW mv AS\nSELECT concat(a, ':', b, ':', c) AS doc_id, * FROM t; -- can exceed 64 bytes\n\n// after\nCREATE MATERIALIZED VIEW mv AS\nSELECT md5(concat(a, ':', b, ':', c)) AS doc_id, * FROM t;","handlingStrategy":"validation","validationCode":"// SQL: cap string pk length at 64 bytes\nSELECT count(*) FROM mv_for_sink WHERE octet_length(doc_pk::varchar) > 64; -- should be 0","typeGuard":"function checkDocId(id) {\n  if (typeof id === 'string' && Buffer.byteLength(id, 'utf8') > 64) {\n    throw new RangeError('turbopuffer doc id must be <= 64 bytes');\n  }\n  return id;\n}","tryCatchPattern":"try {\n  await sink.write(row);\n} catch (e) {\n  if (String(e).includes('exceeds 64 bytes')) {\n    deadLetter.push({ row, reason: 'doc-id-too-long' });\n  } else { throw e; }\n}","preventionTips":["Hash long natural keys (md5/sha256) before using them as doc IDs","Prefer Int64/Serial primary keys for Turbopuffer sinks","Enforce a 64-byte length check on string keys at ingestion"],"tags":["sink","turbopuffer","value-out-of-range","primary-key","runtime"],"backgroundTag":"value-out-of-range","analyzedSha":"6469eb736d691e8e9b8a419a57edd6429ca77417","analyzedAt":"2026-09-11T21:06:21.487Z","contentChangedAt":"2026-09-11T21:06:21.487Z","schemaVersion":2},"datasetVersion":"2026-09-14T16:17:12.679Z"}