Configuration and building

Introduction

The starknet indexer is developed with usage of modular architecture (flow-based architecture) which was implemented in our indexer SDK. The main idea of the modular approach is building data workflows by modules. More information about flow-based architecture can be found on wiki. The indexer consists of 2 modules: indexer and gRPC server. gRPC server is subscribed on notifications from indexer. This approach allows for sufficient flexibility in implementation and high performance. Also system contains Postgres as a storage and Hasura as a GraphQL API. The system is configurable with config.

Indexer

Indexer is module which receives raw data from sequencer node, parses it and stores to database.

Configuration

The indexer may be configurable in section indexer in config.

indexer:
  name: ${INDEXER_NAME:-dipdup_starknet_indexer}
  sequencer:
    feeder_gateway: ${STARKNET_SEQUENCER_FEEDER_GATEWAY:-https://alpha-mainnet.starknet.io/feeder_gateway}
    gateway: ${STARKNET_SEQUENCER_GATEWAY:-https://alpha-mainnet.starknet.io/gateway}
    requests_per_second: ${STARKNET_SEQUENCER_RPS:-2}
  node:
    url: ${STARKNET_NODE_URL}
    requests_per_second: ${STARKNET_NODE_RPS:-5}
  threads_count: ${INDEXER_THREADS_COUNT:-10}
  start_level: ${INDEXER_START_LEVEL:-0}
  timeout: ${INDEXER_REQUEST_TIMEOUT:-10}
  class_interfaces_dir: ${INDEXER_CLASS_INTERFACES_DIR:-./interfaces/}
  bridged_tokens_file: ${INDEXER_BRIDGED_TOKENS_FILE:-mainnet.json}
  cache_dir: ${INDEXER_CACHE_DIR:-/etc/starknet}

Config field	Environment variable	Default value	Description
name	INDEXER_NAME	dipdup_starknet_indexer	Name of indexer module. Using in State model
sequencer.feeder_gateway	STARKNET_SEQUENCER_FEEDER_GATEWAY	https://alpha-mainnet.starknet.io/feeder_gateway	Sequencer feeder gateway URL
sequencer.gateway	STARKNET_SEQUENCER_GATEWAY	https://alpha-mainnet.starknet.io/gateway	Sequencer gateway URL
sequencer.requests_per_second	STARKNET_SEQUENCER_RPS	2	Count requests per second which allowed by sequener node
node.url	STARKNET_NODE_URL		Base URL to node RPC. If object node is empty all requests route to sequencer node
node.requests_per_second	STARKNET_NODE_RPS	5	Count requests per second which allowed by sequener node
threads_count	INDEXER_THREADS_COUNT	10	Count of indexer threads which receives data from sequencer
start_level	INDEXER_START_LEVEL	0	Level from which indexer starts to receive data. Used for debugging only
timeout	INDEXER_REQUEST_TIMEOUT	10	Timeout in seconds for receiving data from node
class_interfaces_dir	INDEXER_CLASS_INTERFACES_DIR	./interfaces/	Directory contained description of contract interfaces (e.g.ERC20)
bridged_tokens_file	INDEXER_BRIDGED_TOKENS_FILE	mainnet.json	Path to file containes bridged tokens of starknet
cache_dir	INDEXER_CACHE_DIR	/etc/starknet	Directory which used for caching node responses. Uses only for debugging.

gRPC

gRPC is used for live notifications from indexer. It can be configurable via dipdup.yml in grpc section.

grpc:
  bind: ${GRPC_BIND:-127.0.0.1:7779}

Config field	Environment variable	Default value	Description
bind	GRPC_BIND	127.0.0.1:7779	bind address on which grpc serves

Database

Configuration of database via dipdup.yml:

database:
  kind: postgres
  host: ${POSTGRES_HOST:-db}
  port: ${POSTGRES_PORT:-5432}
  user: ${POSTGRES_USER:-dipdup}
  password: ${POSTGRES_PASSWORD:-changeme}
  database: ${POSTGRES_DB:-starknet}

Config field	Environment variable	Default value	Description
kind	-	postgres	Kind of using database. It's always postgres now.
host	POSTGRES_HOST	db	Host where database is served.
port	POSTGRES_PORT	5432	Port where database is served.
user	POSTGRES_USER	dipdup	Database user
password	POSTGRES_PASSWORD	changeme	Database password
database	POSTGRES_DB	starknet	Database name

Hasura

Hasura configuration in dipdup.yml:

hasura:
  url: http://${HASURA_HOST:-hasura}:${HASURA_PORT:-8080}
  admin_secret: ${ADMIN_SECRET:-changeme}
  select_limit: 100
  allow_aggregation: false
  source:
    name: default
    database_host: ${HASURA_POSTGRES_HOST}
    use_prepared_statements: true
    isolation_level: read-committed
  rest: true

Config field	Environment variable	Default value	Description
url	HASURA_HOST, HASURA_PORT	http://hasura:8080	URL where Hasura is hosted
admin_secret	ADMIN_SECRET	changeme	Admin password for access to Hasura
select_limit	-	100	Limit to request data from GraphQL
allow_aggregation	-	false	Allow aggregation in GraphQL
allow_aggregation	-	false	Allow aggregation in GraphQL
source.name	-	default	Name for Hasura source
source.database_host	HASURA_POSTGRES_HOST	-	Database host for Hasura. Uses for debugging
source.use_prepared_statements	-	true	If set to true the server prepares statement before executing on the source database
source.isolation_level	-	read-committed	The transaction isolation level in which the queries made to the source will be run with
rest	-	true	Allow REST API instead of GraphQL

Building

The indexer can be built by docker compose or make. To build the indexer you should set .env file with all needed wnvironment variables. For example:

LOG_LEVEL=info
POSTGRES_HOST=127.0.0.1
POSTGRES_DB=starknet-goerli2
HASURA_HOST=127.0.0.1
HASURA_POSTGRES_HOST=db
INDEXER_THREADS_COUNT=2
INDEXER_REQUEST_TIMEOUT=20
INDEXER_CLASS_INTERFACES_DIR=../../build/interfaces
INDEXER_BRIDGED_TOKENS_FILE=../../build/bridged_tokens/goerli2.json
INDEXER_CACHE_DIR=/home/artem/.goerli2-starknet
STARKNET_SEQUENCER_FEEDER_GATEWAY=https://alpha4-2.starknet.io/feeder_gateway
STARKNET_SEQUENCER_GATEWAY=https://alpha4-2.starknet.io/gateway
STARKNET_SEQUENCER_RPS=3
STARKNET_NODE_URL=<URL_HERE>
STARKNET_NODE_RPS=5

To build with docker compose clone repository and run docker-compose

https://github.com/dipdup-io/starknet-indexer.git
docker-compose up -d --build

To build with make clone repository and run make

git clone https://github.com/dipdup-io/starknet-indexer.git
make indexer