Improving performance

This page contains tips that may help to increase indexing speed.

Configure table indexes

Postgres indexes are internal tables that Postgres can use to speed up data lookup. A database index acts like a pointer to data in a table, just like an index in a printed book. If you look in the index first, you will find the data much quicker than searching the whole book (or — in this case — database).

You should add indexes on columns often appearing in WHERE clauses in your GraphQL queries and subscriptions.

DipDup ORM uses BTree indexes by default. To set index on a field, add db_index=True to the field definition:

from dipdup import fields
from dipdup.models import Model


class Trade(Model):
    id = fields.BigIntField(primary_key=True)
    amount = fields.BigIntField()
    level = fields.BigIntField(db_index=True)
    timestamp = fields.DatetimeField()

Perform heavy computations in a separate process

For the most deferred calculations you can use built-in job scheduler. However, DipDup jobs are executed in the same asyncio loop as the rest of the framework, so they can affect indexing performance.

See Services for information on how to run multiple services in DipDup.

Reducing disk I/O

Indexing produces a lot of disk I/O. During development you can store the database in RAM. By default DipDup uses in-memory SQLite database dropped on exit. Using tmpfs instead allows you to persist the database between process restarts until the system is rebooted. By default, tmpfs is mounted on /tmp with a size of 50% of RAM. The following spells are for Linux, but on macOS the process should be similar.

# Make sure tmpfs is mounted
$ df -h /tmp
Filesystem      Size  Used Avail Use% Mounted on
tmpfs            16G  1.0G   16G   7% /tmp

# You can change the size of tmpfs without unmounting it
$ sudo mount -o remount,size=64G,noatime /tmp

# But make sure that you have enough swap for this
$ free -h
               total        used        free      shared  buff/cache   available
Mem:            30Gi        16Gi       3,1Gi       1,3Gi        11Gi        12Gi
Swap:           31Gi       6,0Mi        31Gi

# Update database config to use tmpfs
$ grep database -A2 dipdup.yaml
database:
  kind: sqlite
  path: /tmp/uniswap.sqlite

# After you've done indexing, move the database from RAM to disk
$ mv /tmp/uniswap.sqlite ~/uniswap.sqlite

Commands above were checked on Linux, but on macOS the process should be similar.