F.A.Q.

This page contains answers to the most frequently asked questions about DipDup.

General

What is DipDup?

DipDup is a Python framework for building indexing applications. It allows you to index data from various blockchains and other sources, process it, and store it in a database. DipDup is designed to be fast, efficient, and easy to use.

Why DipDup?

• Declarative configuration: DipDup separates business logic from indexing mechanics through a YAML-based configuration file, making it easy to understand and modify your indexing rules without diving into code.
• Type-safe development: Generated typeclasses provide type hints for smart contract data, enabling IDE autocompletion and catching errors before runtime.
• Multi-chain support: DipDup supports multiple blockchains including EVM-compatible chains, Starknet, Substrate and Tezos, allowing you to build cross-chain applications with a unified API.
• Hassle-free deployment: Deploy DipDup on any machine with Python and PostgreSQL/SQLite. Docker integration simplifies cloud, on-premises, or local deployments.
• Developer experience: Rich CLI tooling, comprehensive documentation, and a helpful community make development smooth and efficient.
• Monitoring and observability: Built-in Prometheus metrics, Sentry integration, crash reports and detailed help messages make it easy to track application health and diagnose issues.
• Free and open source: DipDup is licensed under MIT license, giving you the freedom to use and modify it for any purpose.

What hardware do I need to run DipDup?

DipDup can run on any amd64/arm64 machine that runs Python. Aim for a good single-threaded and disk I/O performance.

Actual RAM requirements can grow significantly depending on the number and complexity of indexes, the size of internal queues and caches, and CachedModel usage.

Indexing

How to index similar but not identical contracts as a single entity?

Multiple contracts can provide the same interface but have a different storage structure. Examples are ERC20/ERC721/ERC1155 standard tokens on Ethereum and FA1.2/FA2 ones on Tezos. If you try to use the same typename for them, indexing will fail because of the storage mismatch. However, you can modify typeclasses manually. Edit types/<typename>/storage.py file and comment out fields leaving only the ones used in your index (common for all contracts with the same interface).

class ContractStorage(BaseModel):
    class Config:
        extra = Extra.ignore

    common_ledger: dict[str, str]
    # unique_field_foo: str
    # unique_field_bar: str

Don't forget Extra.ignore Pydantic hint, otherwise, storage deserialization will fail. To restore the original typeclass, remove the modified file and run dipdup init again. You can also add --force flag to overwrite all ABIs and typeclasses.

How to use off-chain datasources?

DipDup provides convenient helpers to process off-chain data like market quotes or IPFS metadata. Follow the tips below to use them most efficiently.

• Do not perform off-chain requests in handlers until necessary. Handlers need to be as fast as possible not to block the database transaction. Use hooks instead, enriching indexed data on-demand.
• Use generic http datasource for external APIs instead of plain aiohttp requests. It makes available the same features DipDup uses for internal requests: retry with backoff, rate limiting, Prometheus integration etc.
• Database tables that store off-chain data can be marked as immune, to speed up reindexing.

How to process indexes in a specific order?

Indexes of all kinds are fully independent. They are processed in parallel, have their own message queues, and don't share any state. It is one of the essential DipDup concepts, so there's no "official" way to manage the order of indexing.

Avoid using sync primitives like asyncio.Event or asyncio.Lock in handlers. Indexing will be stuck forever, waiting for the database transaction to complete.

Instead, save raw data in handlers and process it later with hooks when all conditions are met. For example, process data batch only when all indexes in the dipdup_index table have reached a specific level.

Database

How to perform database migrations?

Using migrations is not recommended. DipDup architecture is designed to be simple and predictable. It uses a single database schema for all indexes, and any changes to the schema require a full reindex to ensure data consistency. Consider using SQL scripts instead as described below.

If want to proceed with migration tools, DipDup provides integration with aerich. See Migrations section for details.

You may want to disable the schema hash check in config. Alternatively, call the schema approve command after every schema change.

advanced:
  reindex:
    schema_modified: ignore

I get schema_modified error, but I didn't change anything

DipDup compares the current schema hash with the one stored in the database. If they don't match, it throws an error. If models were not modified, most likely the reason is incorrect model definitions. e.g. if you define a timestamp field like this…

timestamp = fields.DatetimeField(default=datetime.utcnow())

…schema will be different every time you run DipDup, because datetime.utcnow() is evaluated on a module import.

$ dipdup schema export > schema.sql
$ dipdup schema export > same-schema.sql
$ diff schema.sql same-schema.sql 
116c116
<     "timestamp" TIMESTAMP NOT NULL  DEFAULT '2023-04-19T21:16:31.183036',
---
>     "timestamp" TIMESTAMP NOT NULL  DEFAULT '2023-04-19T21:16:36.231221',

You need to make the following change in models.py:

<     timestamp = fields.DatetimeField(default=datetime.utcnow())
>     timestamp = fields.DatetimeField(auto_now=True)

We plan to improve field classes in future releases to accept callables as default values.

Why am I getting decimal.InvalidOperation error?

If your models contain DecimalFields, you may encounter this error when performing arithmetic operations. It's because the value is too big to fit into the current decimal context.

class Token(Model):
    id = fields.TextField(primary_key=True)
    volume = fields.DecimalField(decimal_places=18, max_digits=76)
    ...

The default decimal precision in Python is 28 digits. DipDup tries to increase it automatically by guessing the value from the schema. It works in most cases, but not for really big numbers. You can increase the precision manually in config.

advanced:
  decimal_precision: 128

Don't forget to reindex after this change. When decimal context precision is adjusted you'll get a warning in the logs.

WARNING  dipdup.database      Decimal context precision has been updated: 28 -> 128

How to modify schema manually?

Drop an SQL script into sql/on_reindex/ directory. It will be executed after the Tortoise schema initialization. For example, here's how to create a Timescale hypertable:

CREATE EXTENSION IF NOT EXISTS timescaledb CASCADE;

ALTER TABLE swap DROP CONSTRAINT swap_pkey;
ALTER TABLE swap ADD PRIMARY KEY (id, timestamp);
SELECT create_hypertable('swap', 'timestamp', chunk_time_interval => 7776000);

If you want to modify existing schema and know what you're doing, put the idempodent (i.e. can be executed multiple times without changing the result) in sql/on_restart/ directory and call dipdup_approve function inside.

ALTER TABLE "event" ADD COLUMN IF NOT EXISTS "timestamp" TIMESTAMP NOT NULL;

SELECT dipdup_approve('public');

Package

What is the symlink in the project root for?

DipDup project must be a valid discoverable Python package. Python searches for packages in site-packages and the current working directory. This symlink is a hack that allows project root to serve as a working directory without requiring a subdirectory. It's created only when dipdup init is executed from the project root. While this symlink trick should be harmless, if it interferes with any of your tools, use DIPDUP_NO_SYMLINK=1 environment variable to disable this behavior.

Maintenance

How to update DipDup?

Run the following command in the project root:

make update

It will update both CLI tool and Python package.

What's the difference between uv/pip/Poetry/others?

tl;dr: Just use uv for everything.

For historical reasons, Python package management is a mess. There are multiple tools and approaches to manage Python dependencies. pip is a general-purpose package manager. It's simple and robust, but only covers basic functionality. For a full-fledged project, you need to use a tool to handle virtual environments, lock files, dependency resolution, publishing, etc. Some of the most popular tools are: uv, Poetry, PDM, Hatch and others.

Starting with version 8.3, DipDup uses uv as the default package manager for both CLI installer and project management. This tool is extremely fast, reliable, and replaces all bulk of functionality provided by other tools.

Poetry and PDM integration in DipDup is deprecated and will be removed in future releases. To perform a migration, run the following commands:

rm *.lock
rm -rf .venv
sed -i 's/\(  package_manager: \).*/\1uv/' configs/replay.yaml
dipdup init --force

Miscellaneous

Why it's called DipDup?

DipDup (/dɪp dʌp/) was initially developed as a grant project for the Tezos Foundation. Tezos is one of the few blockchains with its own contract language, completely different from Solidity. This low-level, stack-based language is called Michelson. The name "DipDup" comes from DIP {DUP}, a small Michelson program that duplicates the second element on the stack. While today we support dozens of blockchains, Tezos holds a special place in our project's history.