risingwavelabs/risingwave · error · SinkError::Http

Turbopuffer document id column cannot be null

Error message

Turbopuffer document id column cannot be null

What it means

Every Turbopuffer document needs an ID, which the sink takes from the primary-key column (`pk_index`) in `id_for_row`. If the pk datum is NULL for a row, no document ID can be built and an Http error is raised for the write.

Source

Thrown at src/connector/src/sink/turbopuffer.rs:519

                        )));
                    }
                    Some(_) => {
                        return Err(SinkError::Http(anyhow!(
                            "unexpected namespace_column type, expected varchar"
                        )));
                    }
                };
                validate_namespace(namespace)?;
                Ok(format!("{}/v2/namespaces/{}", self.base_url, namespace))
            }
        }
    }

    // Turbopuffer document IDs are unsigned 64-bit integers, UUIDs, or strings up to 64 bytes.
    // RisingWave UUID IDs can be represented with varchar.
    fn id_for_row(&self, row: &impl Row) -> Result<DocumentId> {
        let datum = row.datum_at(self.pk_index).ok_or_else(|| {
            SinkError::Http(anyhow!("Turbopuffer document id column cannot be null"))
        })?;
        match datum {
            ScalarRefImpl::Int16(value) => Ok(document_id_from_i64(value as i64)),
            ScalarRefImpl::Int32(value) => Ok(document_id_from_i64(value as i64)),
            ScalarRefImpl::Int64(value) => Ok(document_id_from_i64(value)),
            ScalarRefImpl::Serial(value) => Ok(document_id_from_i64(value.into_inner())),
            ScalarRefImpl::Utf8(value) => {
                if value.len() > 64 {
                    return Err(SinkError::Http(anyhow!(
                        "Turbopuffer string document id exceeds 64 bytes"
                    )));
                }
                Ok(DocumentId::String(value.to_owned()))
            }
            _ => Err(SinkError::Http(anyhow!(
                "Turbopuffer document id column must be an integer or varchar"
            ))),
        }

View on GitHub (pinned to 6469eb736d)

Solutions

1. Ensure the pk column is NOT NULL upstream; drop or repair rows with NULL keys.
2. Pick a non-nullable column as the sink's primary key.
3. Add upstream filtering to skip NULL-key rows.
4. Re-ingest the offending rows with valid keys after fixing the source.

Example fix

// before
CREATE MATERIALIZED VIEW mv AS SELECT a, b FROM t; -- pk 'a' nullable

// after
CREATE MATERIALIZED VIEW mv AS
SELECT * FROM t WHERE a IS NOT NULL; -- then sink pk='a'

// SQL: assert the pk column is non-null before sinking
SELECT count(*) FROM mv_for_sink WHERE doc_pk IS NULL; -- should be 0

function getDocId(row) {
  const id = row.doc_pk;
  if (id == null) throw new TypeError('document id (pk) must not be null');
  return id;
}

try {
  await sink.write(row);
} catch (e) {
  if (String(e).includes('document id column cannot be null')) {
    deadLetter.push({ row, reason: 'null-document-id' });
  } else { throw e; }
}

When it happens

Trigger: Writing a row whose primary-key column is NULL to a Turbopuffer sink; NULL pk values arriving from a source or MV where the pk is nullable.

Common situations: LEFT JOINs nulling out pk columns upstream; nullable source columns declared as pk by mistake; data corrections/backfills inserting rows with NULL keys.

Related errors

• Turbopuffer namespace_column cannot be null
• Turbopuffer string document id exceeds 64 bytes
• Turbopuffer sink requires exactly one primary_key column
• Turbopuffer document id column must be an integer or varchar
• Primary key not defined for upsert bigquery sink (please def

AI-assisted analysis of risingwavelabs/risingwave@6469eb736d (2026-09-11). Data as JSON: /api/errors/21b2e759cb3cffac. Report an issue: GitHub.