Remake input data placement upon site list changes

[amaltaro]

Impact of the new feature
MSTransferor

Is your feature request related to a problem? Please describe.
This is a sub-task of this meta-issue: #8323
Towards providing a feature that allows workflow site lists to be changed while workflows are active in the system.

Describe the solution you'd like
Whenever the site lists of an active workflow changes, we need to revisit the input data placement rules - if any - and update their RSE expressions accordingly. That will involve at least the following:

1. accessing the workflow transfer document to find out the rule ids
2. if a site was added to the SiteBlacklist, that site needs to be removed from the rule ids RSE expression
3. if a site was added to the SiteWhitelist, that site needs to be added to the rule ids RSE expression
4. the new rule ids need to be persisted in the transfer document, replacing those rules that are no longer valid.

This can be implemented either:
a) synchronous with the status transition, but it might make the client HTTP request just too expensive (ReqMgr2 + Globalqueue + Rucio change all in a single client call)
b) asynchronous, but then we need a mechanism to flag/identify workflows that need data re-placement.

Describe alternatives you've considered
There are a few ways to implement this feature, like:
a) trigger a new data placement with the new site lists
b) once the new rule creation is sucessful, we could delete the previous rule superseded
c) similar to a), but instead of making a new rule, we could consume the rule ids already persisted in the database (via MSTransferor/MSMonitor) and update their RSE expression accordingly.

Hasan made the following observation for case c): "You cannot update (update-rule) the rse expression of a rule and keep the same rule id. You can change the rse expression by "moving" (move-rule) a rule which creates a new rule."

Additional context
NOTE that the best would be to let MSTransferor take care of this data re-placement, such that it considers everything (campaign configuration, pileup configuration, rse quota, etc).

[amaltaro]

PS: based on implementation details, we might have to break this down into 2 or 3 issues.

[todor-ivanov]

@amaltaro as a logical continuation of the effort on delivering the dynamic SiteLists changes, I decided it would be good to take this issue as well and start looking into/thinking about the eventual logic while we go through the review process of #12099 . Please let me know if you think this was not a good idea and I will step down .....