Multiple services

DipDup provides several ways to run multiple services as a part of a single application. This is useful for running background tasks, heavy computations, or any other long-running processes that should not block the main DipDup service.

This table summarizes the available methods:

Job scheduler

DipDup has a built-in job scheduler that allows you to run tasks on schedule or in background.

Here's a minimal configuration for long-running service:

hooks:
  service:
    callback: service

jobs:
  service_daemon:
    hook: service
    daemon: true

Run dipdup init after adding this configuration to create a callback stub in hooks directory:

DipDup jobs are executed in the same process, thread and asyncio event loop as the main DipDup service, and use the same database connection. You can use ctx to communicate with indexer and access its state.

CLI command

Another approach is to implement a custom CLI command that runs in a separate process, but still uses the same DipDup configuration and environment variables. This is useful for running heavy computations or background tasks that do not need to be part of the main indexing process.

First, create a new file cli.py in the project root directory:

from contextlib import AsyncExitStack

import click
from dipdup.cli import cli, _cli_wrapper
from dipdup.config import DipDupConfig
from dipdup.context import DipDupContext
from dipdup.utils.database import tortoise_wrapper


@cli.command(help='Run heavy calculations')
@click.option('--key', help='Example option', required=True)
@click.pass_context
@_cli_wrapper
async def heavy_stuff(ctx, key: str) -> None:
    config: DipDupConfig = ctx.obj.config

    async with tortoise_wrapper(
        url=config.database.connection_string,
        models=f'{config.package}.models',
    ):
        ...

Note that ctx argument is click.Context object, not a dipdup.context.DipDupContext! If you need deeper integration with DipDup, use the following code:

from dipdup.dipdup import DipDup
from dipdup.context import DipDupContext


dipdup: DipDup = DipDup(ctx.obj.config)
dipdup_ctx: DipDupContext = dipdup._ctx

Then create a __main__.py file to make the module executable:

from dipdup_indexer.cli import cli

if __name__ == '__main__':
    cli(prog_name='dipdup', standalone_mode=True)

Then use python -m dipdup_indexer instead of dipdup as an entrypoint. Now you can call heavy-stuff like one of DipDup commands. dipdup.cli:cli group handles arguments and config parsing, graceful shutdown, and other boilerplate.

python -m dipdup_indexer heavy-stuff --key value

If your new command is a long long-running service, you need to update Dockerfile and Compose manifests. Update the deploy/Dockerfile to use the new entrypoint:

ENTRYPOINT ["python", "-m", "dipdup_indexer"]

For Docker Compose, add a new service to deploy/compose.yaml:

services:
  dipdup_service:
    build:
      context: .
      dockerfile: deploy/Dockerfile
      command: ["heavy-stuff", "--key", "value"]
    ...

Run make up to update the stack.

Isolated package

You can also create a separate package with its own dipdup.yaml configuration file and reuse only the necessary parts of the project. Useful for monorepos with multiple DipDup projects.

New project directory can be either a sibling of the main project or a subdirectory. In this example, we will create a nested package dipdup_indexer/service

# Create a new package with its own DipDup config
dipdup new --name service

# Move to the service directory
cd service
make install
source .venv/bin/activate