Auto-scaling capabilities

[fkorotkov]

With platforms like Scaleway that has an API and provide not only macOS infrastructure but a more generic one. We can add builtin functionality that will scale Orchard workers up and down based on the demand.

There are couple things to consider:

• Orchard Auto-Scaler can configure workers automatically via SSH. Our Ansible roles are not that hard to replicate in scripts.
• Auto-scaler should be aware of the 24 hours minimum limit of that we can rent hosts for.

I think the initial algorithm version should be super simple. There should be just couple configurations:

• maximumWorkers/minimumWorkers - range to scale workers within.
• upscaleSpeed - percentage to which upscale in one iteration. For example, if there are 10 unschedulable VMs in the queue and speed in 50% then the auto-scaler will create 3 nodes to fit 5 VMs.
• upscaleInterval - duration to wait before the next upscale attempt.
• Similar downscaleSpeed / downscaleInterval
• Something else?

In case of auto-scaling we should also consider changing scheduling algorithm for maximum utilization -- prefer to schedule as many VMs on the same host as possible before going to the next one.