Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Fix] Speed up "--resume" #1548

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

chtzs
Copy link

@chtzs chtzs commented May 23, 2024

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers. By the way, if you're not familiar with how to use pre-commit to fix lint issues or add unit tests, please refer to Contributing to OpenMMLab.

Motivation

In the current version, "--resume" will make engine loading data like regular training, but discard all of this data before reaching the specified iteration. Therefore, the time required for resuming will not be much faster than starting a new training session.

Modification

Added a new method called skip_iter , which skips data without triggering data loading by calling the _next_index in the built-in iterator of DataLoader

Checklist

  1. Pre-commit or other linting tools are used to fix the potential lint issues.
  2. The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
  3. If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDetection or MMPretrain.
  4. The documentation has been modified accordingly, like docstring or example tutorials.

@CLAassistant
Copy link

CLAassistant commented May 23, 2024

CLA assistant check
All committers have signed the CLA.

@zhouzaida
Copy link
Collaborator

Hi @chtzs , thanks for your contribution. There is a similar issue about the --resume feature. Maybe we can continue the dicussion at #1520

@zhouzaida zhouzaida linked an issue May 23, 2024 that may be closed by this pull request
@fpshuang
Copy link

插眼

@hyalvin
Copy link

hyalvin commented Jun 17, 2024

hi, when will the PR being merged? Is it possible to release this fix in 0.10.5?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] Speed up the resume process of IterBased loop
5 participants