Skip to main content
Version: Next

Embeddings Evaluation Tool

The evaluation tool scores every configured text representation on semantic search and duplicate detection (experimental features), so a decision between representations (or models, or quantization settings) rests on measured numbers instead:

bin/console bpt:embeddings:search-eval [options]

The command is read-only towards the search index: it embeds the labeled queries (its only writing is those query embeddings landing in the short-TTL query cache pool, where they expire on their own) and runs kNN searches. Everything it scores must therefore already be in the index — see the prerequisites.

Prerequisites​

  1. Configure embeddings — models and usages under pimcore_backend_power_tools.embeddings (semantic_text_search / object_dedup for objects, image_search / image_dedup for assets), with the inference endpoint reachable.

  2. Enable evaluation mode — by default, only the primary text representation is stored; the comparison needs all of them:

    pimcore_backend_power_tools:
        embeddings:
            evaluation_mode: true

  3. Restart your messenger workers (e.g. bin/console messenger:stop-workers, letting the supervisor restart them) — long-running workers keep the container they booted with and would otherwise still write the old format set.

  4. Recreate the index so the mapping contains the vector fields of every representation: bin/console generic-data-index:update:index -r, then run bin/console bpt:embeddings:reindex --reconcile — the blue-green swap copies documents natively, so elements the queue has not rewritten yet are missing the new fields until reconciled.

  5. Generate and index the embeddings for the corpus (the command prints which mode is active):

    bin/console bpt:embeddings:reindex
    bin/console messenger:consume pimcore_generic_data_index_queue \
        pimcore_backend_power_tools_embedding_queue --time-limit=300   # repeat until both are empty

    An element without vectors is scored as a miss, so an incomplete backfill understates every representation. --debug shows the indexed element count; --dump-text=<id> shows what was embedded for one element.

  6. Write the label files (format below) against elements that exist in the corpus — expected ids missing from it are dropped and reported, not silently scored.

info

While evaluation_mode is enabled, the indexing pipeline stores one nested vector (plus hash and MinHash siblings) per text representation for every element. See Cleaning up once evaluation is finished.

How it works​

  1. The corpus is loaded — all data objects of one class (--type=object) or all image assets (--type=asset) — and their text is composed with the same extractors and format composers production indexing uses.
  2. Every labeled query is embedded once and reused across all representations.
  3. Each query runs a kNN search against the production index — the same index, mapping and quantization the live search uses. Nothing is re-indexed, and nothing is written; evaluating the real index keeps the measurement as clean as possible.
  4. The ranking is compared to the label file's expectations and aggregated per representation.

Duplicate detection is scored per anchor element: exact duplicates are matched by the stored content hash (byte-identical composed text), near-duplicates by vector candidates re-ranked with the stored MinHash signature. The reported number is the rank-1 rate — how often the true duplicate is the top-ranked candidate, which is what a "likely duplicates, most likely first" review screen shows. It is deliberately not a score cut-off: a true duplicate scores ~1.00 and a near-identical variant ~0.99, so no single cosine threshold separates them.

Label files​

Search labels — one JSON array per file, each entry one query with the element ids it must find (primary marks the single best answer, optional):

[
    { "query": "red italian sports car", "relevant": [12, 16, 97], "primary": [97] }
]

Dedup labels — one entry per anchor with its known duplicates (and optionally merely-similar records, which must appear on page 1 but do not count towards rank-1):

[
    { "anchor": 9, "duplicates": [1168], "similar": [10, 11] }
]

Options​

OptionDescription
--typeobject (data objects, text→text) or asset (images, text→image + image duplicates). Default: object
--classDataObject class name (--type=object only). Default: Car
--search-labelsCSV of search label JSON files path — each file is scored as its own section
--dedup-labelsDedup label JSON file path
--queryDiagnostic: embed this text and print the top hits per representation (skips scoring)
--dump-textDiagnostic: print the composed text for one object id per representation; all prints a JSON content-hash report of the whole corpus
--debugAlso print per-step progress, per-query breakdown tables and the measured index footprint

Examples​

# score all text representations on search + duplicate detection
bin/console bpt:embeddings:search-eval --type=object --class=Car \
    --search-labels=labels/search.json --dedup-labels=labels/dedup.json

# score all text representations on multiple search + duplicate detection
bin/console bpt:embeddings:search-eval --type=object --class=Car \
    --search-labels=labels/search.json,labels/search-attribute-labels.json --dedup-labels=labels/dedup.json

# image search + image duplicate detection
bin/console bpt:embeddings:search-eval --type=asset \
    --search-labels=labels/image-search.json --dedup-labels=labels/image-dedup.json

# what exactly was embedded for object 9?
bin/console bpt:embeddings:search-eval --class=Car --dump-text=9

# qualitative check of a single query
bin/console bpt:embeddings:search-eval --class=Car --query="red sports car"

Reading the results​

 ------------------ ------------- ---------------- ----------------- ---------- ----------- ------------------
  format             search acc.   duplicate acc.   latency avg/max   vec/item   size/item   est. @ 1M items
 ------------------ ------------- ---------------- ----------------- ---------- ----------- ------------------
  values-only        48%           100%             13 / 20 ms        1.00       0.1 KB      91.6 MB + 2.86 GB disk
  key-value          37%           100%             13 / 24 ms        1.00       0.1 KB      91.6 MB + 2.86 GB disk
  position-labeled   41%           100%             12 / 18 ms        1.00       0.1 KB      91.6 MB + 2.86 GB disk
 ------------------ ------------- ---------------- ----------------- ---------- ----------- ------------------
  • search acc. — of the correct answers that fit on page 1 (top 10), how many are actually there. Every query can reach 100% regardless of how many answers it has.
  • duplicate acc. — the rank-1 rate described above.
  • size/item — derived from vector size × vectors per item; a floor excluding index overhead. --debug adds the engine's measured figures.

On the demo dataset (~300 Car objects with nomic_text_v2, 338 image assets with siglip2, binary quantization) the keyless values-only representation scored best on attribute search — 48% against 37% / 41% for the labelled formats, as shown above — and the exact content hash reached 100% recall at 100% precision.

Absolute scores depend entirely on the catalogue, the label files and the metric, so they are only a reference point for the method. The full set of measurements behind the bundle's defaults, each with the conditions it was measured under, is in Defaults and Benchmarks — including why the vector-based duplicate signals are reported as a range rather than a single number.

Cleaning up the comparison representations​

Once evaluation is finished, OpenSearch and Elasticsearch cannot remove a field from an existing mapping, so the cleanup is a reindex:

  1. Set evaluation_mode: false (or remove the key — that is the default). From now on, mapping, generation and storage cover only the primary representation. The flag alone does NOT remove the already-stored comparison vectors — they just stop being updated; only the next step reclaims their space. Restart your messenger workers so none keeps writing with the old configuration.
  2. Recreate the index — bin/console generic-data-index:update:index -r. The blue-green rebuild creates the new mapping without the comparison fields; the remaining representation's vectors repopulate from the durable vector cache table as the queue drains, so nothing is re-embedded.
  3. Optionally run bin/console bpt:embeddings:reindex --reconcile afterward to re-enqueue any element whose vector write-back was lost during the index swap.

The dropped representations' vectors vanish with the recreation — the new mapping omits their fields — so no manual cleanup is needed there; their now-orphaned cache rows age out on their own through eviction, since they can never be matched again.