CLIP Model Evaluation Results

CLIP Model Evaluation Results

Report date: 2026-08-12

This report compares the product-behavior results produced by RawCullFB for:

• OpenAI CLIP ViT-B/32 at 224 × 224; and
• OpenCLIP DataComp ViT-B/32-256 using the datacomp_s34b_b86k checkpoint.

The runs follow the product-behavior portion of Evaluating CLIP Models. They cover the canonical 77 semantic queries and a complete all-pairs image-similarity pass over 453 indexed images.

The reports show that both integrations completed cleanly, neither model shows semantic collapse, and both produce coherent image-neighbourhood structures. DataComp is modestly faster and more consistent across paraphrases. Those are promising sanity-check results, but they are not labeled accuracy metrics.

No final release-model selection can be made from these two reports alone. Source-framework parity evidence, pooled relevance labels, a calibrated labeled similarity benchmark, and full operational measurements are still required by the evaluation procedure.

1. Evidence analyzed

The complete model fingerprints recorded in the reports are:

OpenAI
clip:clip-vit-base-patch32_float16_static:openai/clip-vit-base-patch32:clip-vit-base-patch32_float16_static.aimodel:0.4:directory-tree-sha256-v1:a71ceca116e13b334b0679c95bd85d7ecda14fd68bd7a3ea84b83c286005f45b

DataComp
clip:ViT-B-32-256-datacomp_s34b_b86k_float16_static:mlfoundations/open_clip:ViT-B-32-256-datacomp_s34b_b86k_float16_static.aimodel:0.4:directory-tree-sha256-v1:6a3639a2049b8a4ea23fe04c3083e199a4f505433f7c8bd0748b3c8d4fcb1572

The fingerprints are distinct, as required. Each result set therefore identifies a separate model asset and embedding space.

This analysis did not receive the run’s environment record, catalog hashes, query hash, model metadata, parity outputs, or relevance-label CSV. Matching catalog paths and image counts support comparability, but do not independently prove that every catalog byte and software revision was unchanged between the two run times.

2. Report-integrity checks

For 453 images, the number of unique unordered pairs is:

453 × 452 / 2 = 102,378

The reported pair count is therefore complete for both models. There is no evidence of a partial query run, partial similarity run, incompatible vector, or accidental reuse of one model fingerprint for both reports.

3. Semantic sanity-check methodology

The following report-derived diagnostics were calculated independently for each model:

• distinct images returned at rank one across all 77 queries;
• the most frequently reused rank-one image;
• unique images appearing anywhere in the 77 top-50 result sets;
• mean rank-one minus rank-two score margin;
• mean shared images between the top ten of every pair of unrelated and related queries;
• mean shared top-ten images within the five three-query paraphrase groups;
• first-query latency; and
• warmed mean, median, and P95 latency, excluding the first query.

These are collapse, diversity, consistency, and operational diagnostics. They do not establish whether an image is relevant to a query. Relevance must be judged from the image itself using the pooled labeling procedure in the main evaluation guide.

4. Semantic retrieval results

4.1 Diversity and hub behavior

The most frequently returned rank-one image for OpenAI is 62480371.jpg, selected for four queries. The most frequent DataComp winner is 2132732266.jpg, also selected for four queries.

Both results are consistent with healthy retrieval diversity. The established OpenAI sanity reference in the evaluation procedure is 53 distinct top-one images and a maximum hub frequency of 4/77; this run reproduces those exact values. DataComp slightly increases rank-one and top-50 coverage without creating a larger semantic hub.

The score-margin values may be compared between ranks from the same model, but not treated as calibrated confidence or compared directly as accuracy between models. The two models produce different embedding spaces and score distributions.

4.2 Paraphrase consistency

Across all 2,926 pairs of queries, the mean number of shared images in two top-ten result lists is low, as expected:

Paraphrases are materially more consistent than unrelated prompts for both models. This is an important sanity check: changing the wording while retaining the intent produces strongly overlapping results, without causing unrelated queries to collapse onto a common result pool.

The group-level mean shared top-ten counts are:

DataComp has higher top-ten overlap in all five groups. Its largest advantage is on the sharp-portrait group. This is the clearest positive signal for DataComp in the current unlabeled reports.

Top-one identity is less stable than top-ten membership for both models. That is not necessarily a failure: several images can have near-equal relevance, and a small score change can reorder the first few results. Graded relevance labels and nDCG@5 are needed to distinguish harmless reordering from a real quality difference.

4.3 Agreement between models

The models return the same rank-one image for 19 of 77 queries, or 24.7%. Their top-ten lists contain an average of 4.69 identical images, or 46.9% of a ten-result list.

Concrete objects, scenes, colors, and attributes show the strongest agreement. Composition and subjective photographic judgments show the weakest agreement, followed by count, spatial, and negative-language prompts.

The strongest top-ten agreement includes:

• a yellow flower: 10/10 shared results;
• a lake surrounded by mountains: 9/10;
• food on a plate: 8/10;
• dog on a beach: 8/10; and
• canine at the seaside: 8/10.

The most model-sensitive prompts include:

• a subject positioned using the rule of thirds: 0/10 shared results;
• a photograph with strong leading lines: 1/10;
• a candid emotional moment: 1/10;
• a reflection of a person in a mirror: 1/10;
• sharp portrait: 1/10; and
• a photograph where the subject is not sharply focused: 1/10.

Disagreement is not itself proof that either result is wrong. It identifies the queries that should receive the closest attention during blind relevance labeling.

5. Query latency

DataComp is consistently but only modestly faster in this report. The first query is the largest difference. Warmed latency is effectively in the same operational class for both models.

These timings are useful observations, not the complete operational benchmark required by the evaluation procedure. The reports do not establish that 30 unmeasured warm-up queries were run, and they do not contain P99 latency, model load time, indexing throughput, peak memory, energy impact, bundle size, or index size.

6. Image-similarity results

DataComp completes the all-pairs similarity pass 19 ms, or 2.2%, faster. It produces two more mutual top-one pairs and a broader set of unique top-one targets. OpenAI has a lower maximum target-reuse count.

OpenAI’s distances are numerically lower throughout the distribution. This does not mean that OpenAI is more accurate: distance and cosine values are calibrated differently in the two embedding spaces. A threshold selected for one model must never be copied to the other model.

As an additional, non-gating diagnostic, 50 obvious two-file groups were identified from shared numeric filename prefixes under images/. This gives 100 eligible anchors. Both models retrieved the apparent partner at the same rates:

This heuristic is encouraging, but filenames are not ground-truth labels. It must not replace the labeled pair set prescribed by the evaluation procedure: exact duplicates, re-encodes, edits, crops, burst frames, same-subject images, hard negatives, and unrelated negatives.

7. What the reports establish

The two reports establish that:

1. RawCullFB completed the same 77-query and 453-anchor workload for both distinct model fingerprints.
2. No incompatible image pairs were encountered.
3. Neither model exhibits obvious semantic collapse or a universal top-one image.
4. Both models distinguish paraphrases from unrelated queries.
5. DataComp has stronger paraphrase overlap in this run.
6. DataComp has slightly broader retrieval diversity.
7. DataComp is modestly faster for semantic queries and the all-pairs similarity pass.
8. The models disagree most on composition, subjective photographic quality, counting, spatial relations, and negation.

The reports do not establish that:

• either converted bundle matches its source framework numerically;
• one model has higher semantic Precision@1, Precision@5, MRR, or nDCG@5;
• one model has a better duplicate-detection ROC-AUC or precision-recall AUC;
• a shared image-similarity threshold is valid;
• all catalog bytes, query bytes, software revisions, and index identities were frozen and archived; or
• either model has acceptable indexing throughput, memory, energy, and storage costs for release.

8. Provisional decision table

9. Recommendation and next steps

DataComp is the more promising candidate in this unlabeled RawCullFB run. It is more paraphrase-consistent, slightly more diverse, and modestly faster, while retaining healthy similarity-neighbourhood behavior.

The current evidence is not sufficient to prefer DataComp for release. The evaluation procedure explicitly requires a material labeled-quality improvement before replacing the established OpenAI control. If labeled quality is effectively tied, OpenAI remains the simpler established choice and DataComp may remain experimental.

Complete the following work before making the model decision:

1. Archive the environment, source revisions, model metadata, fingerprints, catalog manifest, catalog hashes, and semantic-query hash for this run.
2. Attach the OpenAI Hugging Face and DataComp OpenCLIP parity results, proving text cosine of at least 0.999 and end-to-end fixture cosine of at least 0.998.
3. Pool and blind-label the union of both models’ top five results for every query using relevance grades 0, 1, and 2.
4. Calculate Precision@1, Precision@5, MRR, and nDCG@5 overall and by query category.
5. Prioritize manual review of composition, subjective-quality, count, spatial, and negation prompts because those categories differ most strongly.
6. Build the labeled image-pair benchmark and calculate ROC-AUC, PR-AUC, false-positive rate, false-negative rate, and a model-specific threshold.
7. Measure indexing throughput, peak memory, P99 latency, bundle size, index size, and energy impact under controlled conditions.
8. Revisit the provisional decision using the completed decision table.

Until those steps are complete, the defensible conclusion is:

Both models pass the RawCullFB report-integrity and semantic-collapse sanity checks. DataComp shows better paraphrase consistency and small operational advantages, but the release choice remains open pending parity evidence and blinded relevance and similarity labels.