Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"Parallelize" the dashboard build #21

Open
CasperWA opened this issue Jan 21, 2021 · 2 comments
Open

"Parallelize" the dashboard build #21

CasperWA opened this issue Jan 21, 2021 · 2 comments
Labels
CI Issue or PR to do with the Continuous Integration (CI) enhancement New feature or request

Comments

@CasperWA
Copy link
Member

CasperWA commented Jan 21, 2021

It takes more than half an hour to build the dashboard currently. And this is without some expected providers to come online.
One way to minimize the runtime for the build would be to "parallelize" it, if possible.

@CasperWA CasperWA added enhancement New feature or request CI Issue or PR to do with the Continuous Integration (CI) labels Jan 21, 2021
@CasperWA CasperWA changed the title "Parallelize" the dashboard build "Parallelilize" the dashboard build Jan 21, 2021
@CasperWA CasperWA changed the title "Parallelilize" the dashboard build "Parallelize" the dashboard build Jan 25, 2021
@ml-evs
Copy link
Member

ml-evs commented Jan 26, 2022

I added some simple multiprocessing-based parallelism (over providers) in #82, but unfortunately github actions only gives you 2 cores to play with so there is limited benefit (wish I had checked this first...).

On my machine it brings the dashboard build down to ~10 minutes. Another approach would be to split the dashboard build into multiple CI jobs that read from a static providers file (would need to submodule the providers repo), but this would be much more work.

@ml-evs
Copy link
Member

ml-evs commented Jan 26, 2022

Since the network should be the bottleneck, I am now testing whether overloading the 2 cores with 8 processes still leads to a reasonable speed-up.

Another alternative to the above is adding parallelism/async to the validator itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
CI Issue or PR to do with the Continuous Integration (CI) enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants