Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Excruciatingly slow ERC-20 token transfers export #499

Open
eresque opened this issue Oct 1, 2024 · 0 comments
Open

Excruciatingly slow ERC-20 token transfers export #499

eresque opened this issue Oct 1, 2024 · 0 comments

Comments

@eresque
Copy link

eresque commented Oct 1, 2024

I am using ethereum-etl to parse all specific token transfers from start to end block. As a node I'm running a local snap node with all the receipts being saved parameter (--txlookuplimit 0).

To test this instruments work I've decided to parse USDT token (0xdAC17F958D2ee523a2206206994597C13D831ec7) transfers from 20_500_000 block to 20_505_000 block.

The issue I'am having that it works really slow. I am talking taking about 15±0.5 seconds to parse all the transfers.
The official docs state that:

You can tune --batch-size, --max-workers for performance.

But tunning --max-workers parameters actually gives me a slight decrease in speed; --batch-size set to 4096 increases speed to 12.4±0.3 seconds, but it's further tweaking worsens the result.

For my original task I woult need to parse millisons of blocks, so the current speed is unacceptable.

The command I'm running:

ethereumetl export_token_transfers --start-block 20500000 --end-block 20505000 --provider-uri http://127.0.0.1:8545 --output token_transfers.csv  --tokens 0xdAC17F958D2ee523a2206206994597C13D831ec7

I've read this Medium article: How to Export the Entire Ethereum Blockchain to CSV in 2 hours for $10, but I can replicate, since I can't get access to AWS services.

Am I facing a hardware bottleneck or is there something I can do to make it faster?

Would much appreciate any suggestions!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant