Refactor func load_model to class ModelLoader #1909
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* organize comman var into member var of class ModelLoader * split operations in load_model into separate member funcs * refactor cfg.load_in_Xbit to kwarg
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
part of #1758
This PR refactor the func
load_modelinsrc/axolotl/utils/models.pyinto a classModelLoader. Different member functions of classModelLoaderare separated according to their features, and all the member vars ofModelLoaderare shared in these funcs. Moreover, this refactoring make the pipeline of model loading more clearly.TODO:
Mainly changes are listed here:
ModelLoaderload_modelinto separate member funcscfg.load_in_Xbitto kwargThe UML of
ModelLoader:Motivation and Context
Why is this change required?
As the models loaded in Axolotl support more and more features, the func
load_modelis huge now. And this results in confusion about variable changes when abstracting part of funcload_model(#1758 (review)). Refactoringload_modelwill optimize the code structure and facilitate stable evolution when introducing more features in the future.How has this been tested?
open_llama_3b_v2model, and here comes the screenshot of inferencing:However, I don't have access to Ampere or newer GPU, thus I cannot pass the UT on my local machine. It would be nice if all UTs could be tested on CI.