Indexes

Index is a primary DipDup entity connecting the inventory and data handling rules.

Multiple indexes are available for different workloads. Every index is linked to a specific datasource and provides a set of handlers for different kinds of data. Use this table to choose the right index for the task:

Indexes can join multiple contracts considered as a single application. Also, contracts can be used by multiple indexes of any kind, but make sure that they are independent of each other and that indexed data don't overlap.

Handler is a callback function, called when new data has arrived from the datasource. The handler receives the data item as an argument and can perform any actions, e.g., store data in the database, send a notification, or call an external API.

Using templates

Index definitions can be templated to reduce the amount of boilerplate code. To create an index from the template use the following syntax:

templates:
  operation_index_template:
    kind: tezos.operations
    datasources:
      - <datasource>
    ...

indexes:
  template_instance:
    template: operation_index_template
    values:
      datasources:
      - tzkt_mainnet

You can also spawn indexes from templates in runtime; see Spawning in runtime.

Limiting scope

One can optionally specify block levels DipDup has to start and stop indexing at, e.g., there's a new version of the contract, and there's no need to track the old one anymore.

indexes:
  my_index:
    first_level: 1000000
    last_level: 2000000

If last_level is set for all indexes, DipDup will stop indexing when the last one reaches the specified level.

Index templates

Templates allow you to reuse index configuration, e.g., for different networks (mainnet/ghostnet) or multiple contracts sharing the same codebase.

templates:
  my_template:
    kind: tezos.operations
    datasources:
      - <datasource>
    contracts:
      - <contract>
    handlers:
      - callback: callback
        pattern:
          - destination: <contract>
            entrypoint: call

Templates have the same syntax as indexes of all kinds; the only difference is that they additionally support placeholders enabling parameterization:

field: <placeholder>

The template above can be resolved in the following way:

indexes:
  templated_index:
    template: my_template
    values:
      datasources:
      - tzkt_mainnet
      contract: some_dex

Any string value wrapped in angle brackets is treated as a placeholder, so make sure there are no collisions with the actual values. You can use a single placeholder multiple times. In contradiction to environment variables, dictionary keys cannot be placeholders.

An index created from a template must have a value for each placeholder; the exception is raised otherwise. These values are available in the handler context as ctx.template_values dictionary.

Spawning in runtime

You can also spawn indexes from templates in runtime. To begin with, you need to define index templates in the top-level templates section of the config. Then call ctx.add_contract and ctx.add_index methods from any user callback. To achieve the same effect as in snippets from previous paragraph, you can use the following code:

await ctx.add_contract(
    kind='tezos',
    name=originated_contract,
    address=originated_contract,
    typename='registry',
)
await ctx.add_index(
    name=name,
    template='registry_dao',
    values={'contract': originated_contract},
)

The most common use case is to create an index that tracks the originations of contracts originated by a specific contract. We call such indexes index factories.

Pro-tip: you can also use external API as a configuration source, and spawn all necessary contracts/indexes in on_restart system hook.