What causes "Did not find any files matching {file_pattern}" in xai-org/x-algorithm?

List the directory (tf.io.gfile.listdir) and confirm .tfrecord files exist at that exact path

xai-org/x-algorithm · error · ValueError

Did not find any files matching {file_pattern}

Error message

Did not find any files matching {file_pattern}

What it means

build_dataset globs '{dataset_path}/*.tfrecord' with tf.io.gfile.glob and raises ValueError when the result is empty, i.e. the directory holds no TFRecord files (or the path/glob resolves nowhere on the local or remote filesystem).

Source

Thrown at adult-content/dataset_utils.py:51

    if features_as_dict:
        features = {"media_embedding": embedding}
    else:
        features = embedding

    return features, label


def build_dataset(
    dataset_path, embedding_dim, batch_size, do_resample=False, do_repeat=False
):
    file_pattern = f"{dataset_path}/*.tfrecord"
    files = tf.io.gfile.glob(file_pattern)

    random.shuffle(files)

    if not len(files):
        raise ValueError(f"Did not find any files matching {file_pattern}")

    ds = tf.data.TFRecordDataset(files).map(
        lambda x: decode_fn_embedding(x, embedding_dim)
    )
    ds = ds.map(lambda x: preprocess_embedding_example(x, positive_label=1))

    if do_resample:
        ds = ds.apply(resample_fn).map(lambda _, b: (b))

    ds = ds.batch(batch_size=batch_size)

    if do_repeat:
        ds = ds.shuffle(buffer_size=10).repeat()

    return ds

View on GitHub (pinned to 24c60942c5)

Solutions

List the directory (tf.io.gfile.listdir) and confirm .tfrecord files exist at that exact path
Fix dataset_path to the directory that actually contains the shards
If files use a different extension, rename them or adjust the pattern
Re-run the dataset extraction/generation step that should have produced the tfrecords

Example fix

# before
build_dataset('/gs/adult-content/train', ...)
# after
build_dataset('/gs/adult-content/train_shards', ...)  # dir actually containing *.tfrecord

Defensive patterns

Strategy: validation

Validate before calling

files = tf.io.gfile.glob(f"{dataset_path}/*.tfrecord")
assert files, f"no tfrecords under {dataset_path}"

Type guard

def has_tfrecords(path: str) -> bool:
    return len(tf.io.gfile.glob(f"{path}/*.tfrecord")) > 0

Try / catch

try:
    ds = build_dataset(path, ...)
except ValueError as e:
    if 'Did not find any files' in str(e):
        path = locate_dataset(); ds = build_dataset(path, ...)
    else:
        raise

Prevention

Assert dataset dirs are non-empty in data-validation CI
Name shards with the canonical .tfrecord extension
Have generation jobs emit a _SUCCESS marker checked before training

When it happens

Trigger: Calling build_dataset(dataset_path=...) on a directory with no .tfrecord files; wrong dataset_path (typo, missing shard prefix); files present but named differently (.tfrecords, .record); GCS/S3 path with wrong bucket/prefix so glob returns an empty list.

Common situations: Pointing train_model at an extraction step that failed or wrote elsewhere; extension convention mismatch; empty shards directory after a partial upload; calling build_dataset before dataset generation completed.

Related errors

AI-assisted analysis of xai-org/x-algorithm@24c60942c5 (2026-08-28). Data as JSON: /api/errors/c4d2e71b4c24fef9. Report an issue: GitHub.