Indexing Details

Index Structure

All data that is delivered by simple rest endpoints is indexed in search indices. Queries and data delivery takes place directly out of the search index (not from Pimcore database).

For each Datahub configuration separate search indices will be created and updated.

Indexing of data takes places asynchronously with an update queue and the process queue command datahub:simple-rest:process-queue. This command needs to be executed on a regular basis, e.g. every 5 minutes.

Index mapping and queue filling takes place automatically when creating and updating Datahub configurations. In addition, also commands for index management are available (datahub:simple-rest:create-or-update-mapping, datahub:simple-rest:init-index).

The create-or-update-mapping command supports a --force-recreate option that drops the alias and both physical indexes, recreates a fresh index with the current mapping, and re-enqueues all elements. Use this for breaking mapping changes that cannot be migrated by the default blue/green reindex (e.g. flat_object → keyword). The command asks for interactive confirmation; pass --no-interaction to skip the prompt.

When a mapping change cannot be applied with putMapping alone, the bundle performs a blue/green reindex into the other physical index and switches the alias once it succeeds. That reindex is submitted to the cluster as an asynchronous task and then polled, because a synchronous _reindex keeps a single HTTP request open for the whole rebuild — managed clusters behind a proxy terminate such a request long before a large index finishes, which would make the mapping update impossible to complete. Tune the polling budget if a rebuild legitimately takes longer than an hour:

pimcore_data_hub_simple_rest:
    reindex_settings:
        # maximum number of polls before giving up (default: 720)
        max_polls: 720
        # seconds between polls (default: 5) — 720 x 5s = 1 hour
        poll_interval: 5

If a reindex fails, the alias is left pointing at the previous index, so the endpoint keeps serving the old data until a later run succeeds. The half-built target index is reclaimed as well, but only once the reindex task has been cancelled and confirmed stopped - deleting an index a task is still writing into does not stop the task, it simply recreates the index under dynamic mapping. When the cancellation cannot be confirmed the index is kept with a warning in the pimcore_datahub_simplerest log channel, and the next run reclaims it.

Per endpoint, multiple indices are created - one for each dataobject class, one for dataobject folders, one for assets and one for asset folders.

For asset metadata (EXIF, XMP, and IPTC), indexed dynamic objects are used. Turn indexing off for these objects to avoid indexing conflicts, or when the limit of maximum data fields is reached.

Turn indexing off in the indexing_options area of the bundle configuration. The data is then still stored in the index and delivered via the endpoint, but it is not indexed.

Tree Hierarchy Management The indexing process keeps a valid folder structure in the index. Based on workspace settings, a combined parent folder is calculated. This combined parent folder might be a sub folder of the parent folder in the Pimcore folder structure, and all element paths are rewritten to it.

Workspace and data schema settings can also produce missing links in the folder structure. In this case, the indexing process creates virtual folders to fill up these gaps.

For updating whole index structure after changes, multiple runs of datahub:simple-rest:process-queue might be necessary (since additional items might be added to queue during queue processing).

Indexing Options

Via symfony configuration, detailed indexing options can be configured.

Global Options

Define some global options like automatic data type detection for dynamic objects.

pimcore_data_hub_simple_rest:
    indexing_options:
        global_options:

            # Enable numeric detection for dynamic objects (like embedded asset metadata, etc.)
            numeric_detection:    false

            # Enable date detection for dynamic objects (like embedded asset metadata, etc.)
            date_detection:       true

Assets

Define if embedded metadata of assets should be indexed. If enabled, they will be indexed as dynamic objects.

pimcore_data_hub_simple_rest:
    indexing_options:
        assets:

            # Enable indexing for exif data
            enable_exif:          true

            # Enable indexing for xmp data
            enable_xmp:           true

            # Enable indexing for iptc data
            enable_iptc:          true

Number of Shards

By default, the number of shards for indexes is set to 1. Change it for all created indices via default_number, or per index via the index_specific setting.

# Configure number of shards for created indices
pimcore_data_hub_simple_rest:
    indexing_options:
        number_of_shards_config:
        
            # default number is picked if no index specific settings is set
            default_number:       3
        
            # Define number of shards for certain indices. Define index name (without -odd/-even postfix) as key, and number of shards as value.
            index_specific:       
                enterprise_simple_rest__pt_rest__asset: 5
                enterprise_simple_rest__<ENDPOINTNAME>__<asset|DataObjectClass>: 3