More complex model management (multiple models, model reloading etc...)

[bsergean]

⚠️ BEFORE SUBMITTING, READ:

We're excited for your request! However, here are things we are not interested in:

• Decorators.
• Doing the same thing in multiple ways.
• Adding more layers of abstraction... tree-depth should be 1 at most.
• Features that over-engineer or complicate the code internals.
• Linters, and crud that complicates projects.

---

🚀 Feature

Supporting model reloads (when a new version is available) and multiple models.

Motivation

Other servers supports this so to be more attractive that would be a nice feature.

Pitch

Right now it's obvious on how to serve one model, but what if there are multiple ones (and the request (binary, or HTTP arguments) will tell which model should be used).

Alternatives

Run N instances for the N models present at a certain time, but if a new model appear, that won't work.

Additional context

We have an internal C++ server that supports this, torch.serve support that too with I believe what they call an orchestrator.

[bsergean]

Looks like a duplicate of #271