Indexes

An index is a core DipDup entity that connects your data inventory with the rules for handling blockchain data. Indexes define what to track (contracts, events, operations, etc.), where to get the data (datasource), and how to process it (handlers). They are the main building blocks for structuring your indexing logic in DipDup.

Multiple index kinds are available for different blockchains and workloads. Each index is linked to a specific datasource and provides a set of handlers for different kinds of data. Use the table below to choose the right index for your task:

Indexes can join multiple contracts considered as a single application. Contracts can be used by multiple indexes of any kind, but make sure that they are independent of each other and that indexed data don't overlap.

A handler is a callback function triggered when new data arrives from the datasource. Handlers receive the data item as an argument and can perform any action, such as storing data in the database, sending notifications, or calling external APIs.

Handlers are defined in your config and linked to specific patterns (e.g., entrypoints, events, or operations).

Using templates

Index definitions can be templated to reduce the amount of boilerplate code. To create an index from the template use the following syntax:

templates:
  operation_index_template:
    kind: tezos.operations
    datasources:
      - <datasource>
    ...

indexes:
  template_instance:
    template: operation_index_template
    values:
      datasources:
      - tzkt_mainnet

You can also spawn indexes from templates at runtime; see Spawning in runtime.

Limiting scope

One can optionally specify block levels DipDup has to start and stop indexing at, e.g., there's a new version of the contract, and there's no need to track the old one anymore.

indexes:
  my_index:
    first_level: 1000000
    last_level: 2000000

If last_level is set for all indexes, DipDup will stop indexing when the last one reaches the specified level.

Index templates

Templates allow you to reuse index configuration, e.g., for different networks (mainnet/ghostnet) or multiple contracts sharing the same codebase.

templates:
  my_template:
    kind: tezos.operations
    datasources:
      - <datasource>
    contracts:
      - <contract>
    handlers:
      - callback: on_call
        pattern:
          - destination: <contract>
            entrypoint: call

Templates have the same syntax as indexes of all kinds; the only difference is that they additionally support placeholders enabling parameterization:

field: <placeholder>

The template above can be resolved in the following way:

indexes:
  templated_index:
    template: my_template
    values:
      datasources:
      - tzkt_mainnet
      contract: some_dex

Any string value wrapped in angle brackets is treated as a placeholder, so make sure there are no collisions with the actual values. You can use a single placeholder multiple times. In contradiction to environment variables, dictionary keys cannot be placeholders.

An index created from a template must have a value for each placeholder; the exception is raised otherwise. These values are available in the handler context as ctx.template_values dictionary.

Spawning in runtime

DipDup allows you to spawn indexes from templates dynamically at runtime. To do this, define index templates in the top-level templates section of your config. Then, from any user callback, call ctx.add_contract and ctx.add_index:

await ctx.add_contract(
    kind='tezos',
    name=originated_contract,
    address=originated_contract,
    typename='registry',
)
await ctx.add_index(
    name=name,
    template='registry_dao',
    values={'contract': originated_contract},
)

The most common use case is to create an index that tracks the originations of contracts originated by a specific contract. We call such indexes index factories.

Pro-tip: you can also use external API as a configuration source, and spawn all necessary contracts/indexes in on_restart system hook.