Indexing Details

Index Structure

All data that is delivered by simple rest endpoints is indexed in search indices. Queries and data delivery takes place directly out of the search index (not from Pimcore database).

For each Datahub configuration separate search indices will be created and updated.

Indexing of data takes places asynchronously with an update queue and the process queue command datahub:simple-rest:process-queue. This command needs to be executed on a regular basis, e.g. every 5 minutes.

Index mapping and queue filling takes place automatically when creating and updating Datahub configurations. In addition, also commands for index management are available (datahub:simple-rest:create-or-update-mapping, datahub:simple-rest:init-index).

The create-or-update-mapping command supports a --force-recreate option that drops the alias and both physical indexes, recreates a fresh index with the current mapping, and re-enqueues all elements. Use this for breaking mapping changes that cannot be migrated by the default blue/green reindex (e.g. flat_object → keyword). The command asks for interactive confirmation; pass --no-interaction to skip the prompt.

Per endpoint, multiple indices are created - one for each dataobject class, one for dataobject folders, one for assets and one for asset folders.

For asset metadata (EXIF, XMP, and IPTC), indexed dynamic objects are used. Turn indexing off for these objects to avoid indexing conflicts, or when the limit of maximum data fields is reached.

Turn indexing off in the indexing_options area of the bundle configuration. The data is then still stored in the index and delivered via the endpoint, but it is not indexed.

Tree Hierarchy Management The indexing process keeps a valid folder structure in the index. Based on workspace settings, a combined parent folder is calculated. This combined parent folder might be a sub folder of the parent folder in the Pimcore folder structure, and all element paths are rewritten to it.

Workspace and data schema settings can also produce missing links in the folder structure. In this case, the indexing process creates virtual folders to fill up these gaps.

For updating whole index structure after changes, multiple runs of datahub:simple-rest:process-queue might be necessary (since additional items might be added to queue during queue processing).

Indexing Options

Via symfony configuration, detailed indexing options can be configured.

Global Options

Define some global options like automatic data type detection for dynamic objects.

pimcore_data_hub_simple_rest:
    indexing_options:
        global_options:

            # Enable numeric detection for dynamic objects (like embedded asset metadata, etc.)
            numeric_detection:    false

            # Enable date detection for dynamic objects (like embedded asset metadata, etc.)
            date_detection:       true

Assets

Define if embedded metadata of assets should be indexed. If enabled, they will be indexed as dynamic objects.

pimcore_data_hub_simple_rest:
    indexing_options:
        assets:

            # Enable indexing for exif data
            enable_exif:          true

            # Enable indexing for xmp data
            enable_xmp:           true

            # Enable indexing for iptc data
            enable_iptc:          true

Number of Shards

By default, the number of shards for indexes is set to 1. Change it for all created indices via default_number, or per index via the index_specific setting.

# Configure number of shards for created indices
pimcore_data_hub_simple_rest:
    indexing_options:
        number_of_shards_config:
        
            # default number is picked if no index specific settings is set
            default_number:       3
        
            # Define number of shards for certain indices. Define index name (without -odd/-even postfix) as key, and number of shards as value.
            index_specific:       
                enterprise_simple_rest__pt_rest__asset: 5
                enterprise_simple_rest__<ENDPOINTNAME>__<asset|DataObjectClass>: 3