apache/beam · error

varint too long

Error message

varint too long

What it means

ErrVarIntTooLong is a sentinel error from DecodeVarUint64 in the Beam Go coder package. While decoding a varint (as used in the Beam Fn API runner wire protocol), either the shift exceeded 64 bits or the 64th-bit pattern was invalid, meaning the encoded bytes do not represent a valid uint64 varint — the code path treats it as data corruption.

Source

Thrown at sdks/go/pkg/beam/core/graph/coder/varint.go:28

// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.

package coder

import (
	"io"

	"github.com/apache/beam/sdks/v2/go/pkg/beam/core/util/ioutilx"
	"github.com/apache/beam/sdks/v2/go/pkg/beam/internal/errors"
)

// ErrVarIntTooLong indicates a data corruption issue that needs special
// handling by callers of decode. TODO(herohde): have callers perform
// this special handling.
var ErrVarIntTooLong = errors.New("varint too long")

// EncodeVarUint64 encodes an uint64.
func EncodeVarUint64(value uint64, w io.Writer) error {
	ret := make([]byte, 0, 8)
	for {
		// Encode next 7 bits + terminator bit
		bits := value & 0x7f
		value >>= 7

		var mask uint64
		if value != 0 {
			mask = 0x80
		}
		ret = append(ret, (byte)(bits|mask))
		if value == 0 {
			_, err := ioutilx.WriteUnsafe(w, ret)
			return err
		}

View on GitHub (pinned to 12126d8942)

Solutions

1. Check that reader/writer coder versions match between the harness and SDK (no version skew).
2. Verify stream alignment: re-check where decoding starts; a desync makes the next varint read garbage.
3. Ensure custom encoders use Beam's varint format (7 bits per byte, high bit = continuation) matching EncodeVarUint64.
4. If corruption is suspected, add checksums/re-encode at the source and surface the surrounding byte context for debugging.

// Validate varint byte length before decoding custom data
if len(buf) == 0 || countContinuationBytes(buf) > 10 {
    return errors.New("invalid varint encoding")
}

n, err := coder.DecodeVarUint64(r)
if errors.Is(err, coder.ErrVarIntTooLong) {
    // treat as corrupted data: resync stream, log offset, fail or skip record
}

When it happens

Trigger: Decoding a byte stream where a varint has more than 10 continuation bytes, or the final byte sets bit patterns pushing shift past 63 (shift >= 64, or shift == 63 with bits > 1), typically from misaligned or truncated data streams.

Common situations: Corrupted pipeline data between runner harnesses, deserializing coders from the wrong offset after a coder mismatch, hand-rolling varint encoding incompatible with Beam's LEB128-style format, or version-skewed runner/SDK pairs.

Related errors

• failed to marshal custom coder %v
• custom coders must be length prefixed: %+v
• bad encoding function
• bad decoding function
• value %v not encodable with %v

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/c30c1f5d4357abd7. Report an issue: GitHub.