hcengineering/platform · error

Blob size exceeds limit of 30MB

Error message

Blob size exceeds limit of 30MB

What it means

FullTextIndexPipeline.handleBlob refuses to index any attachment whose size exceeds 30MB (30 * 1024 * 1024 bytes). Full-text extraction of very large blobs would exhaust memory/time, so the pipeline throws instead of downloading and converting the blob.

Source

Thrown at server/indexer/src/indexer/indexer.ts:1143

    if (docInfo !== undefined && docInfo.size < 30 * 1024 * 1024) {
      // We have blob, we need to decode it to string.
      const contentType = (docInfo.contentType ?? defaultContentType).split(';')[0]

      const ct = contentType.toLocaleLowerCase()
      if ((ct.includes('text/') && contentType !== 'text/rtf') || ct.includes('application/vnd.github.version.diff')) {
        await this.handleTextBlob(ctx, docInfo, indexedDoc)
      } else if (isBlobAllowed(contentType)) {
        await this.handleBlob(ctx, docInfo, indexedDoc)
      }
    }
  }

  private async handleBlob (ctx: MeasureContext<any>, docInfo: Blob | undefined, indexedDoc: IndexedDoc): Promise<void> {
    if (docInfo !== undefined) {
      const contentType = (docInfo.contentType ?? '').split(';')[0]

      if (docInfo.size > 30 * 1024 * 1024) {
        throw new Error('Blob size exceeds limit of 30MB')
      }
      const buffer = Buffer.concat(
        await ctx.with('fetch', {}, (ctx) => this.storageAdapter?.read(ctx, this.workspace, docInfo._id))
      )
      let textContent = await ctx.with(
        'to-text',
        {},
        (ctx) => this.contentAdapter.content(ctx, this.workspace.uuid, docInfo._id, contentType, buffer),
        {
          workspace: this.workspace.uuid,
          blobId: docInfo._id,
          contentType
        }
      )
      textContent = textContent
        .split(/ +|\t+|\f+/)
        .filter((it) => it)
        .join(' ')

View on GitHub (pinned to 63e28dc964)

Solutions

1. Exclude or skip large blobs from full-text indexing (filter by contentType/size before the pipeline processes them).
2. Raise the limit only if infrastructure allows, and re-evaluate memory headroom: change the 30 * 1024 * 1024 threshold in indexer.ts.
3. Wrap handleBlob in a catch that logs and marks the doc as 'skipped-oversize' instead of failing the whole pipeline.
4. Ensure blob metadata (docInfo.size/contentType) is correct; a wrong match may point at an unrelated huge blob.

Example fix

// before
if (docInfo.size > 30 * 1024 * 1024) {
  throw new Error('Blob size exceeds limit of 30MB')
}

// after
if (docInfo.size > 30 * 1024 * 1024) {
  ctx.warn('skipping oversize blob', { _id: docInfo._id, size: docInfo.size })
  indexedDoc.skipIndex = true
  return
}

const MAX_INDEXABLE_BLOB = 30 * 1024 * 1024
// before indexing (or before upload, if you want to reject early)
if (docInfo != null && docInfo.size > MAX_INDEXABLE_BLOB) {
  skipIndexing(docInfo._id, 'oversize')
}

function isIndexableBlob (docInfo: Blob | undefined | null): docInfo is Blob {
  return docInfo != null && docInfo.size <= 30 * 1024 * 1024
}

try {
  await pipeline.index(ctx, doc)
} catch (err) {
  if (err.message.includes('Blob size exceeds limit')) {
    ctx.warn('skipping oversize blob', { docId: doc._id })
    markSkipped(doc._id, 'oversize')
  } else {
    throw err
  }
}

When it happens

Trigger: Indexing a document whose attached blob (docInfo.size from the blob metadata) is larger than 31457280 bytes — e.g. large videos, CAD files, big PDFs, disk images uploaded to the workspace and picked up by the full-text indexing pipeline.

Common situations: Workspace contains large media uploads that were never meant to be text-indexed, an indexer re-scanning old documents after upload limits were changed, or blob metadata misreporting sizes (or a wrongly matched docInfo).

Related errors

• Cannot create message, group not found: cardId = ${event.car
• Missing response body
• Storage error ${error.error}
• Empty chunk received ${emptyChunkRetries} times for blob ${n
• Adapter for domain ${domain} not found

AI-assisted analysis of hcengineering/platform@63e28dc964 (2026-08-29). Data as JSON: /api/errors/b2ebf0858e2db32d. Report an issue: GitHub.