risingwavelabs/risingwave · error · SinkError::Http

Turbopuffer string document id exceeds 64 bytes

Error message

Turbopuffer string document id exceeds 64 bytes

What it means

Raised in id_for_row when the primary-key value used as Turbopuffer document id is a string longer than 64 bytes. Turbopuffer accepts string document IDs of at most 64 bytes, so rows whose PK string exceeds this limit cannot be written and the write fails at runtime while building the document id.

Source

Thrown at src/connector/src/sink/turbopuffer.rs:528

                Ok(format!("{}/v2/namespaces/{}", self.base_url, namespace))
            }
        }
    }

    // Turbopuffer document IDs are unsigned 64-bit integers, UUIDs, or strings up to 64 bytes.
    // RisingWave UUID IDs can be represented with varchar.
    fn id_for_row(&self, row: &impl Row) -> Result<DocumentId> {
        let datum = row.datum_at(self.pk_index).ok_or_else(|| {
            SinkError::Http(anyhow!("Turbopuffer document id column cannot be null"))
        })?;
        match datum {
            ScalarRefImpl::Int16(value) => Ok(document_id_from_i64(value as i64)),
            ScalarRefImpl::Int32(value) => Ok(document_id_from_i64(value as i64)),
            ScalarRefImpl::Int64(value) => Ok(document_id_from_i64(value)),
            ScalarRefImpl::Serial(value) => Ok(document_id_from_i64(value.into_inner())),
            ScalarRefImpl::Utf8(value) => {
                if value.len() > 64 {
                    return Err(SinkError::Http(anyhow!(
                        "Turbopuffer string document id exceeds 64 bytes"
                    )));
                }
                Ok(DocumentId::String(value.to_owned()))
            }
            _ => Err(SinkError::Http(anyhow!(
                "Turbopuffer document id column must be an integer or varchar"
            ))),
        }
    }

    fn upsert_row(&self, row: &impl Row, id: DocumentId) -> Result<Map<String, Value>> {
        let mut value = self.row_encoder.encode(row)?;
        value.insert(
            "id".to_owned(),
            serde_json::to_value(id).expect("serialize document id"),
        );
        Ok(value)

View on GitHub (pinned to 6469eb736d)

Solutions

1. Shorten the pk upstream, e.g. hash it: `md5(key)`, `sha256(key)` truncated, or use a numeric surrogate key.
2. Use an Int64/Serial pk instead of a long varchar key.
3. Truncate to <=64 bytes only if uniqueness is preserved (not recommended blindly).
4. Validate key lengths at ingestion time before the sink sees them.

Example fix

// before
CREATE MATERIALIZED VIEW mv AS
SELECT concat(a, ':', b, ':', c) AS doc_id, * FROM t; -- can exceed 64 bytes

// after
CREATE MATERIALIZED VIEW mv AS
SELECT md5(concat(a, ':', b, ':', c)) AS doc_id, * FROM t;

// SQL: cap string pk length at 64 bytes
SELECT count(*) FROM mv_for_sink WHERE octet_length(doc_pk::varchar) > 64; -- should be 0

function checkDocId(id) {
  if (typeof id === 'string' && Buffer.byteLength(id, 'utf8') > 64) {
    throw new RangeError('turbopuffer doc id must be <= 64 bytes');
  }
  return id;
}

try {
  await sink.write(row);
} catch (e) {
  if (String(e).includes('exceeds 64 bytes')) {
    deadLetter.push({ row, reason: 'doc-id-too-long' });
  } else { throw e; }
}

When it happens

Trigger: Writing a row whose varchar pk value exceeds 64 bytes (e.g. long concatenations, hashes-as-hex, full URLs used as keys).

Common situations: Using long natural keys (composite keys joined with delimiters, file paths, URLs) as the sink pk; switching from numeric keys to string keys without truncation policy.

Understand the failure class

Background: "value must be between 0 and 1" / "out of range" / "must not be negative" errors: fixing range-validation failures across open-source libraries — this error's family across 42 libraries.

Related errors

• Turbopuffer document id column cannot be null
• Turbopuffer sink requires exactly one primary_key column
• Turbopuffer namespace_column cannot be null
• Turbopuffer document id column must be an integer or varchar
• Primary key not defined for upsert bigquery sink (please def

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/d4341cbf3f344263. Report an issue: GitHub.