{"record":{"id":"cf1b0123db4d0250","repo":"neondatabase/neon","slug":"unsupported-key-decoded-at-lsn","errorCode":null,"errorMessage":"Unsupported key decoded at LSN {}: {}","messagePattern":"Unsupported key decoded at LSN (.+?): (.+?)","errorType":"exception","errorClass":null,"httpStatus":null,"severity":"error","filePath":"libs/wal_decoder/src/serialized_batch.rs","lineNumber":164,"sourceCode":"        for (shard, record) in shard_records.iter_mut() {\n            assert!(record.batch.is_empty());\n\n            let estimate = Self::estimate_buffer_size(&decoded, shard, pg_version);\n            record.batch.raw = Vec::with_capacity(estimate);\n        }\n\n        for blk in decoded.blocks.iter() {\n            let rel = RelTag {\n                spcnode: blk.rnode_spcnode,\n                dbnode: blk.rnode_dbnode,\n                relnode: blk.rnode_relnode,\n                forknum: blk.forknum,\n            };\n\n            let key = rel_block_to_key(rel, blk.blkno);\n\n            if !key.is_valid_key_on_write_path() {\n                anyhow::bail!(\n                    \"Unsupported key decoded at LSN {}: {}\",\n                    next_record_lsn,\n                    key\n                );\n            }\n\n            for (shard, record) in shard_records.iter_mut() {\n                let key_is_local = shard.is_key_local(&key);\n\n                tracing::debug!(\n                    lsn=%next_record_lsn,\n                    key=%key,\n                    \"ingest: shard decision {}\",\n                    if !key_is_local { \"drop\" } else { \"keep\" },\n                );\n\n                if !key_is_local {\n                    if shard.is_shard_zero() {","sourceCodeStart":146,"sourceCodeEnd":182,"githubUrl":"https://github.com/neondatabase/neon/blob/8f60b04da47ffefe0e52bda2440134b42874eb75/libs/wal_decoder/src/serialized_batch.rs#L146-L182","documentation":"While serializing decoded WAL records per shard, `serialized_batch` maps each block to a relation key and validates it with `is_valid_key_on_write_path()`, which requires the key be i128-representable — concretely that the tablespace-OID field fits u16 or equals the sentinel values 0xFFFFFFFF/0x22222222. A violation means the relation lives in a user-defined tablespace, which Neon storage cannot ingest on the write path.","triggerScenarios":"A tenant executes DDL/DML on objects in a user-defined tablespace (`CREATE TABLESPACE ... LOCATION ...` followed by placing objects there); the resulting WAL blocks decode to keys whose field2 (tablespace OID) exceeds 0xFFFF, failing the check at that LSN.","commonSituations":"Migrating legacy postgres workloads that rely on user-defined tablespaces into Neon; running tools that create tablespaces for data layout; test suites exercising tablespace DDL.","solutions":["Drop the user-defined tablespace and move objects to pg_default/pg_global, then retry the writes","Audit for `CREATE TABLESPACE` / `SET default_tablespace` in tenant sessions and remove them","If you control ingestion, skip-and-log these keys only with an explicit policy — silently dropping breaks consistency","Long term, track Neon's tablespace support status rather than working around the check"],"exampleFix":"-- before: tenant uses a user-defined tablespace\nCREATE TABLESPACE fastdisk LOCATION '/mnt/fast';\nCREATE TABLE t(i int) TABLESPACE fastdisk;\n\n-- after: keep objects in the default tablespace\nCREATE TABLE t(i int);","handlingStrategy":"validation","validationCode":"let key = rel_block_to_key(rel, blk.blkno);\nif !key.is_valid_key_on_write_path() {\n    // reject before serialization: typically a user-defined tablespace\n    tracing::warn!(%key, \"key not writable; check tenant for user-defined tablespaces\");\n    continue;\n}","typeGuard":"fn is_writable_key(key: &Key) -> bool {\n    key.is_valid_key_on_write_path()\n}","tryCatchPattern":null,"preventionTips":["Forbid CREATE TABLESPACE and default_tablespace changes on Neon tenants","Audit migrated workloads for tablespace usage before ingesting their WAL","If skipping invalid keys, do it with an explicit, logged policy — silent drops break consistency"],"tags":["rust","neon","wal-decoding","tablespace","pageserver"],"backgroundTag":"user-defined-tablespace-unsupported","analyzedSha":"8f60b04da47ffefe0e52bda2440134b42874eb75","analyzedAt":"2026-08-16T23:39:28.135Z","schemaVersion":2},"datasetVersion":"2026-08-17T04:17:16.089Z"}