docs(proposal): add docs for using cascading deletion

Signed-off-by: chang.qiangqiang <[email protected]>

docs/proposals/migration-rollback-protection/README.md

@@ -0,0 +1,393 @@
+---
+title: Migration Rollback Protection
+authors:
+- "@CharlesQQ"
+reviewers:
+- "@RainbowMango"
+- "@XiShanYongYe-Chang"
+- "@chaosi-zju"
+- "@whitewindmills"
+- "@grosser"
+approvers:
+- "@RainbowMango"
+
+creation-date: 2024-07-01
+
+---
+
+# Migration Rollback Protection
+
+## Summary
+
+<!--
+提供一种联邦资源的删除策略，可供用户选择当删除联邦层工作负载时，是否同步删除成员集群中的工作负载。
+此设置在工作负载迁移场景中特别有用，可以确保在不影响成员集群上运行的工作负载的情况下快速执行回滚。
+-->
+
+Provide a deletion strategy for federated resources, allowing users to choose whether to synchronously delete the workloads in member clusters when deleting workloads at the federation level.
+
+This setting is particularly useful during workload migration scenarios to ensure that rollback can occur quickly without affecting the workloads running on the member clusters.
+
+## Motivation
+
+<!--
+Karmada 系统的当前行为是这样的，当用户删除 Karmada 控制面的资源时，会同步删除成员集群中被分发的资源。当在某些场景下，例如工作负载迁移回滚场景，用户希望能够保留成员集群中的工作负载。
+-->
+
+The current behavior of the Karmada system is that when a user deletes resources from the Karmada control plane, the distributed resources in the member clusters are also deleted synchronously. However, in certain scenarios, such as workload migration rollout scenarios, users may wish to retain the workloads in the member clusters.
+
+### Goals
+
+<!--
+- 提供删除控制面资源时保留成员集群中资源的能力，与此同时，清理 Karmada 系统附加在成员集群资源上的 labels/annotations 等信息。
+-->
+
+- Provide the capability to retain resources in member clusters when deleting control plane resources, while at the same time, clean up labels/annotations and other information attached to member cluster resources by the Karmada system.
+
+### Non-Goals
+
+<!--
+- 为不同成员集群定义不同的资源删除策略。
+- 为 Karmada 联邦资源，例如 cronfederatedhpa、federaredhpa、federatedresourcequota 等资源, 提供保留成员集群中资源的能力。
+- 其他删除策略, 比如保留 Karmada 控制面中的 work 对象。
+- 成员集群中的资源的级联删除控制。
+-->
+
+- Define different resource deletion strategies for different member clusters.
+- Provide the capability to retain resources in member clusters for Karmada federated resources, such as cronfederatedhpa, federatedhpa, federatedresourcequota, etc.
+- Other deletion strategies, such as retaining work objects in the Karmada control plane.
+- Cascading deletion control of resources in member clusters.
+
+## Proposal
+
+### User Stories (Optional)
+
+#### Story 1
+
+<!--
+作为管理员，我希望在将工作负载迁移到 Karmada 的过程中，如果出现了任何意外情况，例如云平台无法发布应用程序或者 Pod 出现了意外问题，为了迅速停止损失，需要通过 Karmada 提供的回滚机制，立即恢复到迁移前的状态。
+-->
+
+As an administrator, I hope that during the process of migrating workloads to Karmada, if any unexpected situations arise, such as the cloud platform being unable to publish the application or the Pod encountering unexpected issues, it is necessary to use the rollback mechanism provided by Karmada to immediately revert to the state before the migration in order to quickly stop the loss.
+
+### Notes/Constraints/Caveats (Optional)
+
+- For resources that are not distributed through PropagationPolicy, such as namespace, it is not possible to specify a deletion policy. Unless the controller for automatic resource propagation is disabled, and users are required to propagate resources through PP (PropagationPolicy) / CPP (ClusterPropagationPolicy).
+- In one policy vs multi resource scene, we can't execute delete policy just by per resource.
+
+### Risks and Mitigations
+
+## Design Details
+
+### Extend the fields of PropagationPolicy/ClusterPropagationPolicy
+
+<!--
+通过扩展 `PropagationPolicy/ClusterPropagationPolicy` API，引入一个新的 bool 字段 `PreserveResourcesOnDeletion`, 改字段会被透传到 `ResourceBinding/ClusterResourceBinding` 以及 work 对象，最后由 execution-controller 根据 work 字段的值决定是否删除资源。
+-->
+
+By extending the `PropagationPolicy/ClusterPropagationPolicy` API, a new bool field `PreserveResourcesOnDeletion` is introduced. The field will be transparently transmitted to `ResourceBinding/ClusterResourceBinding` and the work object. Finally, the execution controller determines the deletion strategy based on the value of the work field.
+
+#### API changes
+
+PropagationPolicy/ClusterPropagationPolicy
+```go
+type PropagationSpec struct {
+    ...
+
+    // PreserveResourcesOnDeletion controls whether resources should be preserved on the
+    // member clusters when the resource template is deleted.
+    // If set to true, resources will be preserved on the member clusters.
+    // Default is false, which means resources will be deleted along with the resource template.
+    //
+    // This setting is particularly useful during workload migration scenarios to ensure
+    // that rollback can occur quickly without affecting the workloads running on the
+    // member clusters.
+    //
+    // Additionally, this setting applies uniformly across all member clusters and will not
+    // selectively control preservation on only some clusters.
+    //
+    // Note: This setting does not apply to the deletion of the policy itself.
+    // When the policy is deleted, the resource templates and their corresponding
+    // propagated resources in member clusters will remain unchanged unless explicitly deleted.
+    //
+    // +optional
+    PreserveResourcesOnDeletion *bool `json:"preserveResourcesOnDeletion,omitempty"`
+}
+```
+
+ResourceBinding/ClusterResourceBinding
+```go
+type ResourceBindingSpec struct {
+    ...
+
+    // PreserveResourcesOnDeletion controls whether resources should be preserved on the
+    // member clusters when the binding object is deleted.
+    // If set to true, resources will be preserved on the member clusters.
+    // Default is false, which means resources will be deleted along with the binding object.
+    // This setting applies to all Work objects created under this binding object.
+    // +optional
+    PreserveResourcesOnDeletion *bool `json:"preserveResourcesOnDeletion,omitempty"`
+}
+```
+
+Work
+```go
+// WorkSpec defines the desired state of Work.
+type WorkSpec struct {
+    ...
+
+    // PreserveResourcesOnDeletion controls whether resources should be preserved on the
+    // member cluster when the Work object is deleted.
+    // If set to true, resources will be preserved on the member cluster.
+    // Default is false, which means resources will be deleted along with the Work object.
+    // +optional
+    PreserveResourcesOnDeletion *bool `json:"preserveResourcesOnDeletion,omitempty"`
+}
+```
+
+#### Controller logic changes
+
+<!--
+detector 需要将 PreserveResourcesOnDeletion 从 PropagationPolicy/ClusterPropagationPolicy 传递到 ResourceBinding/ClusterResourceBinding 中；
+binding-controller 需要将 PreserveResourcesOnDeletion 从 ResourceBinding 传递到 Work 中；
+cluster-resource-binding-controller 需要将 PreserveResourcesOnDeletion 从 ClusterResourceBinding 传递到 Work 中；
+execution-controller 需要根据 Work 的 PreserveResourcesOnDeletion 字段的值，进行资源删除。
+-->
+
+The `detector` needs to pass the `PreserveResourcesOnDeletion` from PropagationPolicy/ClusterPropagationPolicy to ResourceBinding/ClusterResourceBinding.
+
+The `binding-controller` needs to pass the `PreserveResourcesOnDeletion` from ResourceBinding to Work.
+
+The `cluster-resource-binding-controller` needs to pass the `PreserveResourcesOnDeletion` from ClusterResourceBinding to Work.
+
+The `execution-controller` needs to perform resource deletion based on the `PreserveResourcesOnDeletion` field in Work.
+
+#### User usage example
+
+Set the cascade deletion policy to orphan:
+
+```yaml
+apiVersion: policy.karmada.io/v1alpha1
+kind: PropagationPolicy
+metadata:
+  name: nginx-propagation
+spec:
+  resourceSelectors:
+    - apiVersion: apps/v1
+      kind: Deployment
+      name: nginx
+  preserveResourcesOnDeletion: true
+```
+
+#### Q&A:
+
+1. The resource deletion policy of dependent resources and main resources does not force binding.
+
+<!--
+依赖资源和主资源的删除策略不必强制保持一致。
+由于依赖资源可能被多个资源模版共享，在这种情况下很难决策依赖资源的删除策略以哪个删除策略为准; 不强制和主资源绑定，由用户自己决策，灵活性和扩展性更好。
+-->
+
+Since dependent resources may be shared by multiple resource templates, in this case it is difficult to decide which deletion strategy should be used for the dependent resources; it is not forced to be bound to the main resource, and is left to the user to decide, with greater flexibility and scalability.
+
+2. Whether the workload of the member cluster only clears the `karmada.io/managed` label is enough?
+
+Logically, after the label `karmada.io/managed` is cleared, the relationship with karmada is broken.
+
+### Test Plan
+
+TODO
+
+## Alternatives
+
+### Extended by Annotation
+
+#### API changes
+
+<!--
+新增一个 Annotation，用于用户在 Karmada 控制面中的资源模板上增加，key 值为: `resourcetemplate.karmada.io/cascadedeletion`，为了增加扩展性，value 值为 string 枚举类型，当前支持的类型包括：
+- orphan: 保留成员集群中资源，清理 Karmada 系统附加在成员集群资源上的 labels/annotations 等信息。
+当用户不指定该 annotation 时，为系统当前行为：同步删除成员集群中的资源。
+-->
+
+A new Annotation is added for users to include on resource templates in the Karmada control plane, with the key value: `resourcetemplate.karmada.io/cascadedeletion`. To increase extensibility, the value is of the string enumeration type, and currently supported types include:
+- orphan: Retain resources in member clusters and clean up labels/annotations and other information attached to member cluster resources by the Karmada system.
+
+When users do not specify this annotation, the system's current behavior is to synchronously delete resources in member clusters.
+
+#### Controller logic changes
+
+<!--
+用户添加在资源模板上的 `resourcetemplate.karmada.io/cascadedeletion` annotation 会被传播到 `work.spec.workload.manifests` 中，当资源模板被删除时，`execution-controller` 会执行 work 对象删除的逻辑，它能够从 `work.spec.workload.manifests` 中解析出资源模板上的 `resourcetemplate.karmada.io/cascadedeletion` annotation 值，执行如下判断逻辑：
+- 如果不存在目标 annotation，同步删除成员集群中的资源；
+- 如果目标 annotation 值为 `orphan`，保留成员集群中资源，并清理 Karmada 系统附加在成员集群资源上的 labels/annotations 等信息。
+-->
+
+The `resourcetemplate.karmada.io/cascadedeletion` annotation added by users to the resource template will be propagated to `work.spec.workload.manifests`. When the resource template is deleted, the `execution-controller` will execute the logic for deleting the work object. It can parse the value of the `resourcetemplate.karmada.io/cascadedeletion` annotation from `work.spec.workload.manifests` and perform the following judgment logic:
+- If the target annotation does not exist, synchronously delete the resources in the member clusters.
+- If the target annotation value is `orphan`, retain the resources in the member clusters and clean up the labels/annotations and other information attached to the member cluster resources by the Karmada system.
+
+![resource-delete-policy](./statics/resource-delete-policy.png)
+
+#### User usage example
+
+Set the cascade deletion policy to orphan
+
+```yaml
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  annotations:
+    propagationpolicy.karmada.io/name: foo
+    propagationpolicy.karmada.io/namespace: default
+    resourcetemplate.karmada.io/cascadedeletion: orphan
+    ...
+```
+
+<!--
+在这个方法中，还有一个分支想法是，在 Work API 中增加一个 CascadeDeletion 字段，这样就不用解析 `work.spec.workload.manifests` 了。
+-->
+
+In this approach, there is also a branch idea of adding a `CascadeDeletion` field in the Work API, so there is no need to parse `work.spec.workload.manifests`.
+
+Work
+```go
+// WorkSpec defines the desired state of Work.
+type WorkSpec struct {
+    ...
+
+    // CascadeDeletion Declare the cascade deletion strategy. The default value is null, which is equivalent to background.
+    // +optional
+    CascadeDeletion *CascadeDeletionPolicy `json:"cascadeDeletion,omitempty"`
+}
+```
+
+The `binding-controller` needs to set the `CascadeDeletion` field in the Work object according the resource annotation.
+
+The `cluster-resource-binding-controller` needs to set the `CascadeDeletion` field in the Work object according the resource annotation.
+
+The `execution-controller` needs to perform resource deletion based on the `CascadeDeletion` field in Work.
+
+<!--
+> Note: 对于 namespace 资源来说，Karmada 系统中的 namespace-sync-controller 会将用户新建的每一个 namespace 自动分发至成员集群中，系统通过直接生成 work 对象来实现该功能。对于在 work 中新增 API 字段方案，namespace-sync-controller 需要负责处理该字段。
+-->
+
+> Note: For namespace resources, the `namespace-sync-controller` in the Karmada system automatically propagates each new namespace created by users to member clusters, and the system achieves this functionality by directly generating work objects. For the scheme of adding new API fields in the work, the `namespace-sync-controller` needs to be responsible for processing that field.
+
+#### Advantages & Disadvantages
+
+Disadvantages:
+- Using annotations as an API is somewhat informal.
+
+### Extended by adding a new CRD
+
+<!--
+新增一个 CRD 资源，用户通过定义该 CRD 的 CR 资源，来描述目标资源的资源删除策略。
+-->
+
+A new CRD resource is added, through which users define the CR (Custom Resource) of this CRD to describe the resource deletion strategy for the target resource.
+
+#### API changes
+
+```go
+type CascadeDeletionPolicy struct {
+    metav1.TypeMeta   `json:",inline"`
+    metav1.ObjectMeta `json:"metadata,omitempty"`
+
+    // Spec represents the desired cascadeDeletion Behavior.
+    Spec CascadeDeletionSpec `json:"spec"`
+
+    // Status represents the status of cascadeDeletion.
+    // +optional
+    Status CascadeDeletionStatus `json:"status,omitempty"`
+}
+
+type CascadeDeletionSpec struct {
+    // CascadeDeletion Declare the cascade deletion strategy. The default value is null, which is equivalent to background.
+    // +optional
+    CascadeDeletion *CascadeDeletionPolicy `json:"cascadeDeletion,omitempty"`
+    // ResourceSelectors used to select resources.
+    // Nil or empty selector is not allowed and doesn't mean match all kinds
+    // of resources for security concerns that sensitive resources(like Secret)
+    // might be accidentally propagated.
+    // +required
+    // +kubebuilder:validation:MinItems=1
+    ResourceSelectors []ResourceSelector `json:"resourceSelectors"`
+}
+
+// ResourceSelector the resources will be selected.
+type ResourceSelector struct {
+    // APIVersion represents the API version of the target resources.
+    // +required
+    APIVersion string `json:"apiVersion"`
+
+    // Kind represents the Kind of the target resources.
+    // +required
+    Kind string `json:"kind"`
+
+    // Namespace of the target resource.
+    // Default is empty, which means inherit from the parent object scope.
+    // +optional
+    Namespace string `json:"namespace,omitempty"`
+
+    // Name of the target resource.
+    // Default is empty, which means selecting all resources.
+    // +optional
+    Name string `json:"name,omitempty"`
+
+    // A label query over a set of resources.
+    // If name is not empty, labelSelector will be ignored.
+    // +optional
+    LabelSelector *metav1.LabelSelector `json:"labelSelector,omitempty"`
+}
+
+type CascadeDeletionStatus struct {
+    ...
+}
+```
+
+Work
+```go
+// WorkSpec defines the desired state of Work.
+    type WorkSpec struct {
+    // CascadeDeletion Declare the cascade deletion strategy. The default value is null, which is equivalent to background.
+    // +optional
+    CascadeDeletion *CascadeDeletionPolicy `json:"cascadeDeletion,omitempty"`
+
+    ...
+}
+```
+
+#### Controller logic changes
+
+<!--
+binding-controller/cluster-resource-binding-controller 在创建或更新 work 对象的时候, 查询是否存在关联目标资源的 CascadeDeletionPolicy，如果能够找到，将删除策略同步至 Work 对象中。
+execution-controller 根据 Work 对象中的 CascadeDeletion 字段，进行资源删除。
+-->
+
+The `binding-controller`/`cluster-resource-binding-controller` checks for the existence of a `CascadeDeletionPolicy` associated with the target resource when creating or updating the Work object. If found, the deletion policy is synchronized into the Work object.
+
+The `execution-controller` carries out resource deletion based on the `CascadeDeletion` field in the Work object.
+
+#### User usage example
+
+Set the cascade deletion policy to orphan:
+
+```yaml
+apiVersion: policy.karmada.io/v1alpha1
+kind: CascadeDeletionPolicy
+metadata:
+  name: foo
+spec:
+  cascadeDeletion: orphan
+  resourceSelectors:
+    - apiVersion: apps/v1
+      kind: Deployment
+      name: foo
+      namespace: default
+```
+
+#### Advantages & Disadvantages
+
+Disadvantages:
+- It increases the learning cost for users and results in an increased number of resources in the Karmada control plane.