Monitoring
It's important to configure your monitoring to know that your indexer is working properly. DipDup provides multiple ways to monitor your indexer:
- Prometheus integration
- Internal database tables and views
- Monitoring API endpoint
- Sentry integration
Monitoring endpoint
You can use API endpoint to get the various indexing stats. First, enable internal API in config:
api: {}
By default, DipDup will listen on 0.0.0.0:46339. Query the /performance endpoint:
curl http://0.0.0.0:46339/performance
Internal tables
DipDup uses a set of internal tables to keep track of indexing stats. See the Internal tables page for details.
Additionally, metrics returned by the monitoring endpoint are stored in the dipdup_meta table in JSON format. You can use dipdup_status database view to get a nice summary.
Via SQL:
sqlite> select * from dipdup_status;
type name level size updated_at
---------- ------------------------- -------- ------ --------------------------------
index eth_usdt_events 8211425 0 2024-09-09 20:33:07.482867+00:00
datasource subsquid 20714000 0 2024-09-09 20:22:38.486122+00:00
queue eth_usdt_events:realtime 0 0 2024-09-09 20:33:04.493736+00:00
queue eth_usdt_events:readahead 0 9171 2024-09-09 20:33:04.493736+00:00
cache model:Holder 0 262144 2024-09-09 20:33:04.493736+00:00
Via Hasura:
curl http://0.0.0.0:8000/api/rest/dipdup_head_status?name=eth_events
You can also create your custom alert endpoints using SQL views and functions; see the SQL scripts page for details.
Sentry
Sentry is an error tracking software that can be used either as a service or on-premise. It dramatically improves the troubleshooting experience and requires nearly zero configuration. To start catching exceptions with Sentry in your project, add the following section in dipdup.yaml config:
sentry:
dsn: https://example.com
environment: dev
debug: False
You can obtain Sentry DSN from the web interface at Settings -> Projects -> <project_name> -> Client Keys (DSN). The cool thing is that if you catch an exception and suspect there's a bug in DipDup, you can share this event with us using a public link (created at Share menu).
Prometheus
DipDup provides integration with the Prometheus monitoring system. To enable the integration and listen on 0.0.0.0:9000, add the following section to the config:
prometheus: {}
The following metrics are exposed under dipdup namespace:
| metric name | description |
|---|---|
dipdup_indexes_total | Number of indexes in operation by status |
dipdup_index_total_sync_duration_seconds | Duration of the last synchronization |
dipdup_index_total_realtime_duration_seconds | Duration of the last realtime queue processing |
dipdup_index_levels_to_sync_total | Number of levels to reach synced state |
dipdup_index_levels_to_realtime_total | Number of levels to reach realtime state |
dipdup_index_handlers_matched_total | Index total hits |
dipdup_datasource_head_updated_timestamp | Timestamp of the last head update |
dipdup_datasource_rollbacks_total | Number of rollbacks |
dipdup_http_errors_total | Number of HTTP errors |
Also, DipDup exposes the following metrics for compatibility with Subsquid Cloud:
| metric name | description |
|---|---|
sqd_processor_last_block | Level of the last processed block from Subsquid Network |
sqd_processor_chain_height | Current chain height as reported by Subsquid Network |
sqd_processor_archive_http_errors_in_row | Number of consecutive failed requests to Subsquid Network |
Logging
To control the number of logs DipDup produces, set the logging field in config. It can be either a string or a mapping from logger names to logging levels.
# Configure dipdup and package loggers
logging: WARN
# Increase verbosity of some loggers
logging:
dipdup.database: DEBUG
aiosqlite: DEBUG
# Enable ALL logs
logging:
'': DEBUG
By default only dipdup and dipdup_indexer namespace loggers are configured. DipDup loggers are usually named after the corresponding module, e.g., dipdup.database or dipdup.index.
Keep in mind, that excessive logging can affect indexing performance.
If you need your logs in JSON format, use DIPDUP_JSON_LOG=1 environment variable.