Release Notes: 7.1

Welcome, developers!

After maybe-too-long awaited 7.0 stable release we decided to establish a shorter and more predictable release cycle. So, expect new DipDup versions to arrive more frequently, but maybe be less groundbreaking. Today we're releasing DipDup 7.1 with new index kind, CLI options, and lots of small improvements. Let's dive in!

Indexing Tezos token balances

Starting with 7.1 a new index kind tezos.tzkt.token_balances is available. It allows indexing token balances of Tezos contracts compatible with FA1.2 or FA2 standards. This index is similar to existing tezos.tzkt.token_transfers, but provides only the current balances skipping historical data. In cases when you don't need the history of changes, this index is much more efficient.

spec_version: 2.0
package: demo_token_balances

contracts:
  tzbtc_mainnet:
    kind: tezos
    address: KT1PWx2mnDueood7fEmfbBDKx1D9BAnnXitn
    typename: tzbtc

datasources:
  tzkt:
    kind: tezos.tzkt
    url: https://api.tzkt.io

indexes:
  tzbtc_holders_mainnet:
    kind: tezos.tzkt.token_balances
    datasources:
      - tzkt
    handlers:
      - callback: on_balance_update
        contract: tzbtc_mainnet

SQL helpers to manage database schema

Two new SQL functions were added to the schema: dipdup_approve and dipdup_wipe. These helpers are equivalent to schema approve and schema wipe CLI commands, but can be used directly in SQL queries.

SELECT dipdup_approve('public');

-- WARNING: This action is irreversible! All indexed data will be lost!
SELECT dipdup_wipe('public');

Please note that dipdup_wipe function doesn't support preserving immune tables.

DipDup has lots of tools to work with SQL like parameterized scripts, queries, and more. Visit the Advanced SQL page to learn more.

More config env command options

Managing environment variables could be a pain, still, it's an important part of a modern 12-factor application. Try to keep your project config completely environment-agnostic to make deployment easier and ensure secrets are safe. This release brings a few new options to the config env command to make it easier to work with environment variables.

Compose stack definitions in the deploy package directory include environment variables from .env files (generated with config env command without arguments). New --compose, -c flag also allows exporting config variables in Compose format. Copy the necessary strings from the output to the environment section of the service definition and you're good.

$ dipdup config env --compose
services:
  dipdup:
    environment:
      - API_KEY=${API_KEY}
      - API_URL=${API_URL:-https://api.example.com}

--internal, -i flag allows including internal variables (with DIPDUP_ prefix) in the command output. All of them are optional and rarely needed, but handy for debugging. See the Feature Flags page for the full list of internal variables.

$ dipdup config env --internal
DIPDUP_CI=
DIPDUP_DEBUG=
DIPDUP_DOCKER=
DIPDUP_NEXT=
DIPDUP_REPLAY_PATH=
DIPDUP_TEST=

You can notice a new variable here, DIPDUP_DEBUG. Setting it to true will increase logging verbosity (equal to logging: debug in config) and enable some additional checks. Make sure that this flag is set if you're gathering logs for a bug report.

Finally, the --unsafe flag allows exporting environment variables from the current shell session. The short option missing is intentional to prevent accidental usage. Missing example output here is also intentional, because it's, you know, unsafe to share your secrets. Don't do that.

Improved base template

One of DipDup's killer features is that we scaffold everything: package metadata, types and callback stubs, Compose definitions etc. We constantly improve the base template to make it more useful. You can run dipdup new once, answer a few questions get a fully working project. But you also use the init command later to refresh files from the base template in your project.

# First make sure that repo is clean
git status --porcelane

# Refresh only `pyproject.toml` file
dipdup init --force pyproject.toml

# Refresh everything
dipdup init --force --base

# Check what has changed
git diff

This feature is a huge time saver when you maintain multiple projects and want them to be consistent. Or to modify the replay.yaml file and apply changes to the whole project instead of generating a new one or finding and replacing all occurrences manually.

In this release, we have updated linter rules to be more strict. Scripts to run CI routines were also updated. Run pdm run -l to see the full list of available scripts.

$ pdm run -l
╭────────┬───────────┬──────────────────────────────────╮
│ Name   │ Type      │ Description                      │
├────────┼───────────┼──────────────────────────────────┤
│ all    │ composite │ Run all checks                   │
│ format │ composite │ Format code with isort and black │
│ image  │ cmd       │ Build Docker image               │
│ lint   │ composite │ Check code with ruff and mypy    │
╰────────┴───────────┴──────────────────────────────────╯

Inspect the pyproject.toml file to see how these scripts are defined or add your own to automate common tasks.

Here's a basic example of a GitHub Actions workflow to run all checks on every push:

name: Test DipDup project
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true
on:
  push:
    branches:
      - '**'

jobs:
  test:
    name: Test
    runs-on: ubuntu-latest
    steps:
      - name: Install PDM
        run: pipx install pdm

      - name: Set up Python
        uses: actions/setup-python@main
        with:
          python-version: '3.11'
          cache: 'pip'

      - name: Install project dependencies
        run: pdm install

      - name: Run all checks
        run: pdm all

That's all for today. Thanks for reading!

Changes since 7.0

Added

• cli: Added --unsafe, --compose, --internal flags to config env command.
• cli: Added missing short equivalents for options in some commands.
• cli: Relative paths to be initialized now can be passed to the init command as arguments.
• tezos.tzkt.token_balances: Added new index.

Fixed

• cli: Fixed DIPDUP_DEBUG not being applied to the package logger.
• cli: Fixed crash on early Python 3.11 releases.
• project: Update default Docker tag for TimescaleDB HA.
• tezos.tzkt.token_transfers: Fixed filtering transfers by token_id.

See you soon! 👋

DipDup is a free open-source software created by the community and for the community. Join our socials to discuss this release, ask any questions, or participate in development.

Twitter | Discord | GitHub