Config

Developing a DipDup indexer begins with creating a YAML config file of a specific format. DipDup expects it to be located in the project root and named dipdup.yaml. However, you can provide any path with a -c CLI option.

Structure

Config consists of multiple top-level mappings. In the table below they are grouped by sections for convenience, but in the config file, they can be in any order. The only required section is Header.

See Config reference guide for the full list of available options.

Headerspec_version*DipDup project specification; currently "3.0"
package*Python package name
InventorydatabaseDatabase configuration
contractsContracts that need to be indexed
datasourcesDatasources to gather information from
IndexesindexesIndex definitions
templatesAlso index definitions, but with placeholders that make them reusable
HookshooksCallbacks to run manually or by schedule
jobsSchedules for hooks
IntegrationsapiInternal API configuration
hasuraHasura GraphQL Engine configuration
sentrySentry configuration
prometheusPrometheus configuration
MiscellaneousadvancedTunables that affect framework behavior
customMapping of user-defined values; neither typed nor validated
loggingConfigure logging verbosity

Merging multiple files

DipDup allows you to customize the configuration for a specific environment or workflow. It works similarly to docker-compose anchors but only for top-level sections. If you want to override a nested property, you need to recreate a whole top-level section. To merge several DipDup config files, provide the -c command-line option multiple times:

Terminal
dipdup -c dipdup.yaml -c configs/dipdup.sqlite.yaml run

Use config export and config env commands to check the resulting config used by DipDup.

Environment variables

For the list of variables to configure DipDup see this page.

DipDup supports compose-style variable expansion with an optional default value. Use this feature to store sensitive data outside of the configuration file and make your app fully declarative. If a required variable is not set, DipDup will fail with an error. You can use these placeholders anywhere throughout the configuration file.

dipdup.yaml
database:
  kind: postgres
  host: ${POSTGRES_HOST:-localhost}
  password: ${POSTGRES_PASSWORD}

There are multiple ways to pass environment variables to DipDup:

  • Export them in the shell before running DipDup
  • Create the env file and pass it to DipDup with the -e CLI option

For every config file in the deploy project directory, DipDup will create a corresponding .env.default file with all the variables used in the config. Copy it, remove the .default suffix and fill in the values.

deploy/.env.default
POSTGRES_HOST=localhost
POSTGRES_PASSWORD=

You can use dipdup -e .env config export --unsafe command to ensure that env variables resolve as expected, but avoid sharing the output with third parties.

Contract typenames

typename is an alias for the particular contract script, meaning that two contracts sharing the same code can have the same type name. If this field is not set, a contract alias will be used instead.

dipdup.yaml
contracts:
  kusd_dex_mainnet:
    kind: tezos
    address: KT1CiSKXR68qYSxnbzjwvfeMCRburaSDonT2
    typename: quipu_fa12
  tzbtc_dex_mainnet:
    kind: tezos
    address: KT1N1wwNPqT5jGhM91GQ2ae5uY8UzFaXHMJS
    typename: quipu_fa12

If multiple contracts you index have the same interface but different code, see F.A.Q. to learn how to avoid conflicts.

Reindexing

In some cases, DipDup can't proceed with indexing without a full wipe. Several reasons trigger reindexing:

reasondescription
manualReindexing triggered manually from callback with ctx.reindex.
migrationApplied migration requires reindexing. Check release notes before switching between major DipDup versions to be prepared.
rollbackReorg message received from datasource and can not be processed.
config_modifiedOne of the index configs has been modified.
schema_modifiedDatabase schema has been modified. Try to avoid manual schema modifications in favor of SQL scripts.

It is possible to configure desirable action on reindexing triggered by a specific reason.

actiondescription
exception (default)Raise ReindexingRequiredError and quit with error code. The safest option since you can trigger reindexing accidentally, e.g., by a typo in config. Don't forget to set up the correct restart policy when using it with containers.
wipeDrop the whole database and start indexing from scratch. Be careful with this option!
ignoreIgnore the event and continue indexing as usual. It can lead to unexpected side-effects up to data corruption; make sure you know what you are doing.

To configure actions for each reason, add the following section to the DipDup config:

advanced:
  reindex:
    manual: wipe
    migration: exception
    rollback: ignore
    config_modified: exception
    schema_modified: exception

Advanced options

Flags related to the project are set in the advanced section of the config (most likely in dipdup.yaml).

flagdescription
early_realtimeEstablish realtime connection and start collecting messages while sync is in progress (faster, but consumes more RAM).
decimal_precisionOverwrite precision if it's not guessed correctly based on project models.
postpone_jobsDo not start job scheduler until all indexes reach the realtime state.
rollback_depthA number of levels to keep for rollback.
unsafe_sqliteDisable journaling and data integrity checks. Use only for testing.
Help and tips -> Join our Discord
Ideas or suggestions -> Issue Tracker
GraphQL IDE -> Open Playground