How Detection Works

A run sweeps each configured scope in two tiers.

Tier	Finds	How
Exact	Elements whose composed content hashes identically.	A hash comparison. No score — identical is identical.
Fuzzy	Elements that merely look alike.	Each element probes the vector index with its own stored vectors and retrieves its nearest neighbours.

The vector only generates candidates. A separate scorer decides, and it is not the vector distance: MinHash-based Sørensen–Dice similarity for text, cosine for images. The vector is good at "these are in the same neighbourhood"; the scorer answers "how much do they actually agree".

How objects are scored

Text duplicates are scored over the canonical set of field-value elements extracted from each record — essentially the fraction of field values that agree. An exact copy scores 1.0; a record differing in 2 of 14 field values scores about 0.86. The estimate is computed with MinHash over hashed field=value elements.

What counts as an element:

• One element per atomic value.
• Order-immune. Multi-value and relation storage order does not affect the score.
• Normalized. Lowercased, whitespace collapsed, diacritics folded, so Straße and Strasse are the same value.
• Opaque identifiers dropped. Hex hashes of 16+ digits and UUIDs are excluded automatically — two records are not similar because they both have an id.
• Long text bounded. WYSIWYG and textarea fields contribute exactly 8 sketch elements each, drawn from a per-field MinHash over that field's own word shingles. A shared boilerplate description therefore cannot dominate a pair, while near-identical long text still earns partial credit.

How images are scored

Image duplicates are scored by cosine similarity between the two image embeddings. Cosine scores do not live on the same scale as the MinHash scores above — image pairs sit systematically higher — which is why each scope can carry its own floor.

Stored vs. shown — two thresholds, two different costs

Two thresholds gate a fuzzy finding, at two different moments. Mixing them up is the most common misconfiguration, so it is worth being precise:

• Candidate floor gates what is stored, at analysis time. A candidate scoring below it is never written — not hidden, not filtered later: gone.
• Review threshold gates what is shown, at read time. It applies to the group's top score and to each member's score.

That asymmetry decides what a change costs you:

Change	Needs a run?
Raising or lowering the review threshold	No. It is a read-time filter over what is already stored.
Raising the candidate floor	No new run needed, but the next run discards whatever now falls below it.
Lowering the candidate floor	Yes, a full run. Candidates between the old and new floor were never stored. An incremental run cannot recover them; only re-analysing the scope from scratch can.

The bundle enforces the last row: an incremental run started after any effective floor was lowered is refused, and asks for a full run instead.

Rule of thumb: use the review threshold to tune how noisy the review queue is day to day, and treat the candidate floor as the setting you only lower deliberately.

Clustering — what ends up in one group

The probed element always anchors its own cluster, and a pair is stored once, owned by the element with the lower id — so two elements that each probe the other cannot write two rows about the same pair.

The cluster is then capped to max_members_per_group candidates, applied after structural filtering, so a candidate the filter would drop anyway never consumes a slot.

Structural filters — matches that are dropped outright

Some matches are discarded before the score is even relevant, because no data model makes merging them correct:

• Ancestor and descendant — always. A container is never the same record as its content. If one element's path contains the other's, they are never proposed as duplicates, whatever they score.
• Same-class siblings — per class, suppressed by default. A sibling of the reference (or anything in its subtree) that shares both a parent data object and a class is suppressed. This is what keeps product variants out of the review queue: ten colour variants of one car are not ten duplicates.

The sibling rule is keyed per class precisely because one catalogue can contain both shapes. Leave it on for variant-shaped classes; turn it off for classes like news articles, where two same-folder same-class records that look alike are exactly what you want flagged. See Scope tab.

What else narrows the search

Besides the thresholds and the structural filters, detection is bounded by scope and by retrieval settings:

Lever	Effect
Which classes are enabled	Detection is opt-in per class; an embedded class is not swept until you enable it.
Folders	Include and exclude folders for a scope or a single class. They bound references and members in both tiers, see Folder rules.
Exclusion query (PQL)	A PQL query for a single class or for assets, on top of the folder rules. Matching elements are excluded as references and members in both tiers, see Exclusion query (PQL).
Candidates per probe	How many neighbours one probe retrieves. More candidates, more detection time.
Max probe chunks	For long texts, how many of an element's chunk vectors are used as probes. Cost multiplies by this.
Max members per group	How many candidates are kept per reference.

All of them are in Configuration.