apache/beam · warning

content length for object

Error message

content length for object %s was nil

What it means

Size returns this error when HeadObject succeeds but output.ContentLength is unexpectedly nil, so no size can be reported. This is a defensive check against an anomalous S3 response and is rare with real S3. Size returns -1 in this case.

Solutions

1. Retry or query via GetObject and read the body length if HEAD metadata is missing.
2. Check the S3-compatible endpoint implementation (MinIO, mock) for correct Content-Length in HEAD responses.
3. Update the AWS SDK for Go v2 in case of a deserialization bug in older versions.
4. Treat -1/error as 'size unknown' and continue if size is only advisory.

Example fix

// before
n, err := fs.Size(ctx, uri)
// after
n, err := fs.Size(ctx, uri)
if err != nil || n < 0 {
  n = -1 // fall back to streaming length via io.Copy / GetObject
}

n, err := fs.Size(ctx, uri)
if err != nil {
  log.Warnf("size unavailable for %s: %v; falling back to streaming count", uri, err)
  n = -1 // stream via OpenRead + io.Copy counter if exact size is required
}

When it happens

Trigger: HeadObject returns a 200 response whose ContentLength pointer is nil — essentially only when a stubbed/proxied S3-compatible endpoint (mocks, MinIO misbehavior, custom middleware) omits Content-Length.

Common situations: Testing against fake S3 implementations that don't populate all HeadObjectOutput fields; using third-party S3-compatible storage with non-conformant HEAD responses.

Related errors

• error getting metadata for object
• All parts but the last must be larger than
• AWS dependencies are not installed, and no alternative…
• Basepath %r must be S3 path.
• Cannot rename a directory

AI-assisted analysis of apache/beam@12126d8942 (2026-09-13). Data as JSON: /api/errors/86dd7bb61d294c66. Report an issue: GitHub.

Appendix: source

Thrown at sdks/go/pkg/beam/io/filesystem/s3/s3.go:168

	bucket, key, err := parseURI(filename)
	if err != nil {
		return -1, fmt.Errorf("error parsing S3 uri %s: %w", filename, err)
	}

	params := &s3.HeadObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(key),
	}
	output, err := f.client.HeadObject(ctx, params)
	if err != nil {
		return -1, fmt.Errorf("error getting metadata for object %s: %w", filename, err)
	}

	if output.ContentLength != nil {
		return *output.ContentLength, nil
	}

	return -1, fmt.Errorf("content length for object %s was nil", filename)
}

// LastModified returns the time at which the file was last modified.
func (f *fs) LastModified(ctx context.Context, filename string) (time.Time, error) {
	bucket, key, err := parseURI(filename)
	if err != nil {
		return time.Time{}, fmt.Errorf("error parsing S3 uri %s: %v", filename, err)
	}

	params := &s3.HeadObjectInput{
		Bucket: aws.String(bucket),
		Key:    aws.String(key),
	}
	output, err := f.client.HeadObject(ctx, params)
	if err != nil {
		return time.Time{}, fmt.Errorf("error getting metadata for object %s: %v", filename, err)
	}

View on GitHub (pinned to 12126d8942)