Dune-Sync v2

[fleupold]

It would be great to have a more generalised version of the dune-sync client that allows for both

1. Syncing data from Dune to a local DB
2. Syncing data from a local DB to Dune

Given that Dune has evolved and no longer recommends uploading data into S3 bucket it ingests, I'd suggest for 1) to mainly happen via the upload_csv functionality (as used in https://github.com/cowprotocol/dune-sync/blob/main/src/sync/app_data.py#L21).

I'd also suggest the new version to be completely domain agnostic in that all required information about the source and target data as well as queries can be provided separately (and thus packaged wherever dune-sync is deployed). The fact that for instance at the moment the app_hashes sql is checked into this repo, while the configuration of the sync job happens in the infrastructure repository is a smell imo.

I would suggest something like this as the general configuration shema:

[[datasources]]
name = "postgres"
type = "postgres"
url = "postgresql://host:port/?user=&password="

[[datasources]]
name = "dune"
type = "dune"
apikey = "xyz"

[[jobs]]
from = "dune" # referencing datasource name
to = "postgres" # referencing datasource name
query = 40000123 # referencing dune query id
target = "table" # referencing postgres target table
truncate = false # if table should be truncated before

[[jobs.parameters]]
day = "2021-01-01"
blockchain = "ethereum"

[[jobs]]
from = "postgres"
to = "dune"
query = "SELECT * FROM PRICES" # SQL query string
target = "table_name" # Dune table name
private = false # whether the dune table should be private or not

This way this client becomes a general purpose middleware for any web3 team that wants to bridge data in and out of Dune.