API to cleanup inactive users

[alexeykazakov]

We need some single endpoint which we can call to cleanup resources of an inactive (TBD) user:

• Delete tenant namespaces
• Delete WIT artifacts - spaces, WIs, codebases, ...?
• Clean up che workspaces?
• ...?

Sub tasks:

• Provide an endpoint in WIT to delete corresponding items. So, instead of Auth calling multiple WIT endpoints it will call just one.
• The same as above but for Che if needed.
• New cleanup/delete endpoint in Auth service which calls other services to clean up resources.

[sbose78]

Cleaning up Che workspaces fabric8-services/fabric8-auth#721

[sbose78]

@ibuziuk @l0rd We need an API on the Che side which we could call and that would gracefully clean-up everything on the Che side for a given user.

I see there are a couple of APIs which do something with cleanup and deletion:

1. curl -X DELETE -v 'http://che-server:8080/api/user/:id' -H 'Authorization: Bearer user-token
2. http://che-tenant-maintainer:8080/cleanWorkspaces

Could you help me with a single API to accomplish a complete user cleanup on the Che side? Currently, AFAIK deleting the username-che namespace ( outside of (1) (2) ) causes issues on che-server.

[ibuziuk]

@sbose78 currently there is no API that would both stop / delete all the workspaces and user itself from the Che db. This API should probably be provided by che-starter that plays the role of API adapter but there were no plans to implement it in the nearest future. @l0rd do you think we should prioritize this task ?

Currently, AFAIK deleting the username-che namespace ( outside of (1) (2) ) causes issues on che-server.

Yeah, just deleting namespace might have unpredictable side-effects (at least all running workspaces should be stopped). For now it should be enough just to call:

POST http://che-tenant-maintainer:8080/cleanWorkspaces with the following JSON body

{
"delete-all-workspaces":"true"
}

This would delete all the existing workspaces and after that deleting *-che namespace could be done. @davidfestal maybe you have some comments ?

[sbose78]

POST http://che-tenant-maintainer:8080/cleanWorkspaces with the following JSON body

OK, thanks @ibuziuk . So we'll consider this to be the 'user cleanup' API for a user's Che objects

[kwk]

We need some single endpoint which we can call to cleanup resources of an inactive (TBD) user:

@alexeykazakov may I ask why we need that?

[chmouel]

Related for build: Jenkins proxy deletion API: #4454

[alexeykazakov]

@sbose78 the entry point here should be reg app I think...
We can't clean up everything on OSIO side without deprovisioning OSO users. Otherwise it will be possible to login to the cluster and use it resources directly. If there is no tenant namespaces then nothing will stop them (quota is not reached).

So, our flow should be:

1. Someone/something calls oso reg app API to deprovision user. OSO deprovisions the user and calls Auth to do the rest. We should differ "deprovision and block" from just "deprovision" on both OSO and OSIO sides.
2. Auth calls all relevant services (tenant, wit, etc) to propagate the cleanup.

It means we will need new API in the reg app (if it doesn't exist yet).

WDYT?

[alexeykazakov]

may I ask why we need that?

@kwk to control cluster resources used by OSIO.

[sbose78]

@alexeykazakov , Agreed.

Here's my understanding.

Today:
Ban: Blocks user, deletes namespaces, doesn't delete openshift.io data

Henceforth,

1. Ban : Blocks user, deletes namespaces, doesn't delete openshift.io data. This is a single step process which already exists today.
2. Ban & Deprovision : Blocks user, deletes namespaces, deletes openshift.io data.
   Initially, this should involve the existing Ban and then an API call to Auth to truly delete everything. Later on, this should be an API on reg-app, and also a UI option there. But till we get there, this could be a simple script that does the individual calls.

[michaelkleinhenz]

WIT Epic: https://openshift.io/openshiftio/Openshift_io/plan/detail/1153

[corinnekrych]

@sbose78 Following instructions given by David @davidfestal in as per HK issue to clean workspace, the steps as followed:
1 - login to prod-preview with:

oc login https://api.dsaas-stg.openshift.com —token=XXX

2 - Get pod name for che-tenant-maintainer from dssas preview and port forward:

oc port-forward  che-tenant-maintainer-3-ldkgn 8080

3 - Call the API with OSIO prod-preview token of the user to delete + username in headers:

curl -X POST -d "@curl_clean_workspaces.txt" http://localhost:8080/cleanWorkspaces -H "Authorization: Bearer $KEY" -H "Content-Type: application/json" -H "X-User-Namespace: ckrych-preview-che" -verbose
{"code":0,"message":"","details":"should remove workspace7bx5hsjf2wybyr9k\nshould remove workspacels2b15013jxefd2b\n”}

with curl_clean_workspaces.txt:

{
  "delete-all-workspaces": "true",
  "debug-logs": "true"
}

got a 200 with this error:

{"code":1,"message":"Failure during cleaning of workspace files: Command failed with the following code: 1 and termination message: rm: cannot remove '/pvroot/workspacemynwdmj476xfysuh/projects': Permission denied\nrm: cannot remove '/pvroot/workspacemynwdmj476xfysuh/che-logs/dev-machine': Permission denied\n","details":"removing workspacemynwdmj476xfysuh\n"}

when looking at https://che.prod-preview.openshift.io/dashboard/#/workspaces no workspace is displayed (it looks like it was deleted).

@alexeykazakov @sbose78 not sure how we want to call this API, is it auth-service that is going to call it? as the OSIO token of the user we want to delete is required, is there a way to generate an OSIO token for a given username?

[corinnekrych]

Deletion issue tracked with fabric8-services/che-tenant-maintainer#28

[corinnekrych]

As per discussion in MM quoting shoubhik:
" Let's keep it simple. Let's not enhance the API, we can generate a super shortlived user token in the auth service and then call the che-maintainer. There's no endpoint. But since auth service will be calling the che-maintainer. It can generate a token using the keys. "
we're good to go with this existing che API.
cc @sbose78