-
Notifications
You must be signed in to change notification settings - Fork 2
Project 4
Project 4
- Spawn four instances on Jetstream 1 of medium size
- Install Rancher on one of the instances .
To install Rancher please refer to our peer team Terra's writeup : https://github.com/airavata-courses/terra/wiki/Installing-Rancher---Step--1
Only difference is that it is Jetstream 1 , so you would need to generate the ssh password yourself using
- sudo passwd "username"
Replace your username in the above command.
- Make a k8s cluster : Again refer to https://github.com/airavata-courses/terra/wiki/Step-2:-Setting-up-Kubernetes-cluster-using-Rancher
While adding the nodes to the cluster, choose the calico network
Now that your Rancher and cluster are done, login to the master node .
-
Create Namespaces : Custos, Keycloak, Vault
-
Install helm: https://helm.sh/docs/intro/install/
git clone https://github.com/airavata-courses/DSDummies.git
git checkout custos-deploy-analysis
cd CUSTOS/custos_deploy/
On all the nodes ,
sudo mkdir /bitnami
sudo mkdir /bitnami/mysql
sudo mkdir /bitnami/postgresql
sudo mkdir /hashicorp
sudo mkdir /hashicorp/consul
sudo mkdir /hashicorp/consul/data
chmod 777 -R /hashicorp
Make sure you change the permissions for all directories for hashicorp/consul/data
Also, whichever service you're deploying , always go into that service directory before executing the steps.
cd cert-manager
> kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.8.0/cert-manager.yaml
check output:
kubectl get all -n cert-manager
All the pods should be in running phase. If not , there would be an error so check the kubectl logs to debug the issue.
It should look like this :
> kubectl apply -f issuer.yaml
cd ..
cd keycloak
helm repo add bitnami https://charts.bitnami.com/bitnami
cd postgres
- Create PVs Create three PVs for each mount point /bitnami/postgresql
kubectl apply -f pv.yaml,pv1.yaml,pv2.yaml
Check output :
Then deploy postgresql
helm install keycloak-db-postgresql bitnami/postgresql -f values.yaml -n keycloak --version 10.12.3
Check output :
cd ..
kubectl create -f https://raw.githubusercontent.com/operator-framework/operator-lifecycle-manager/master/deploy/upstream/quickstart/crds.yaml
kubectl create -f https://raw.githubusercontent.com/operator-framework/operator-lifecycle-manager/master/deploy/upstream/quickstart/olm.yaml
git clone https://github.com/keycloak/keycloak-operator
cp operator.yaml keycloak-operator/deploy/
cd keycloak-operator
make cluster/prepare
kubectl apply -f deploy/operator.yaml -n keycloak
cd ..
kubectl apply -f keycloak-db-secret.yaml -n keycloak
kubectl apply -f custos-keycloak.yaml -n keycloak
* Replace hostname in ingress.yaml
kubectl apply -f ingress.yaml -n keycloak
Check output :
user: admin
Get admin password.
kubectl get secret credential-custos-keycloak -o yaml -n keycloak
echo "passwordhere" | base64 --decode
**Store this password , it would be used in further steps **
cd consul
helm repo add hashicorp https://helm.releases.hashicorp.com
- Create directory /hashicorp/consul/data in each of your nodes
sudo chmod 777 -R hashichorp
kubectl apply -f pv.yaml,pv1.yaml
kubectl apply -f storage.yaml
helm install consul hashicorp/consul --version 0.31.1 -n vault --values config.yaml
Check output :
cd vault
helm install vault hashicorp/vault --namespace vault -f values.yaml --version 0.10.0
Change hostname in ingress.yaml
Deploy Ingress
kubectl apply -f ingress.yaml -n vault
At this point, your output should something like this :
-
Follow instructions in UI which is hosted on 443 to generate vault token.
-
Put in 5 and 3 to initialize the keys. It would generate 5 keys, download the keys in the file .
-
In the next step , put the keys in the UI one by one to unseal the vault
After this step, your UI should look like :
The root_token to be used would be found at the end of the file you downloaded.
-
Other way to do (if you don't want to follow through the UI instructions ): Follow step 4 and step 5 from this : https://dwops.com/blog/deploy-hashicorp-vault-on-kubernetes-using-helm/
-
You shall see the vault-0 pod in the vault namespace change from 0/1 to 1/1 Ready phase.
Check output for unsealed vault:
cd mysql
kubectl apply -f pv.yaml,pv1.yaml
Check output :
helm install mysql bitnami/mysql -f values.yaml -n custos --version 8.8.8
Check output :
**Label this for all your nodes in the cluster, replace node_name with all your cluster nodes **
kubectl label nodes node_name custosServiceWorker="enabled"
On master node, execute these steps :
kubectl delete all --all -n ingress-nginx
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v0.44.0/deploy/static/provider/baremetal/deploy.yaml
Now make these code changes on a spare VM on Jetstream:
git clone https://github.com/apache/airavata-custos.git
cd airavata-custos
git checkout develop
Parameters to be added in root pom.xml
UserProfile: dev
Hostname: {Hostname}
SSH_user: {user}
SSH_user_password: {your passwd}
SSH_key: your key path
Keycloak_username: admin
Keycloak_password: keycloak_pass
vault_token: vault_token
MYSQL_user: user
MYSQL_password: pass
docker_repo: {your docker hub repo }
Changes to be made,
-
custos-core-services/utility-services/custos-configuration-service/pom.xml --> change skipped to false
-
custos-core-services/utility-services/custos-configuration-service/resource/*-dev.properties
custos-core-services/utility-services/custos-configuration-service/resource/*-staging.properties
change
iam.server.url=https://{host-name}:30079/auth/
-
Open custos-integration-services/tenant-management-service-parent/tenant-management-service/src/main/java/tasks/TenantActivationTask.java
comment lines 225-249
-
In pom.xml, make sure you change these lines : <vault.scheme>http</vault.scheme> <vault.host>vault.vault.svc.cluster.local</vault.host> <vault.port>8200</vault.port> <vault.uri>http://vault.vault.svc.cluster.local:8200</vault.uri>
-
Create folder custos/artifacts in home directory of master and give 777 permission
-
Create a new Jetstream instance where you shall be running the next steps as it doesn't works on local.
-
On the new instance , execute the following commands :
- sudo apt get install maven
- generate ssh-key with the command as the normal ssh private key doesn't works :
- ssh-keygen -t rsa -b 4096 -m pem
- Login Docker
docker login
- Build code
`mvn clean install -P container`
- Push code images to repo
`mvn dockerfile:push -P container`
- deploy artifacts
mvn antrun:run -P scp-to-remote`
Custos deployed on dev :
Run the following command now once your dev custos pods are running :
helm install cluster-management-core-service /home/ssh_user/custos/artifacts/cluster-management-core-service-1.1-SNAPSHOT.tgz -n keycloak
- Delete the following services step by now (which are deployed on dev):
iam-admin-core-service(On Master node) :
helm uninstall iam-admin-core-service -n custos
Make these code changes in root pom.xml
- <spring.profiles.active>staging</spring.profiles.active>
cd iam-admin-core-service/
sudo mvn clean install -P container
sudo mvn dockerfile:push -P container
sudo mvn antrun:run -P scp-to-remote
identity-core-service(on master node)
helm uninstall identity-core-service -n custos
cd identity-core-service/
sudo mvn clean install -P container
sudo mvn dockerfile:push -P container
sudo mvn antrun:run -P scp-to-remote
Custos deployed with 2 staging and rest dev pods :
- Enable new engines named "secret" select v1 and "resourcesecret" also with v1. Post Request to register tenant https://{hostname}:30079/tenant-management/v1.0.0/oauth2/tenant
{
``"client_name":"dsdummiesproj",``
``"requester_email":"[email protected]",``
``"admin_username":"umang",``
``"admin_first_name":"Umang",``
``"admin_last_name":"Sharma",``
``"admin_email":"[email protected]",``
``"contacts":["[email protected]","[email protected]"],``
``"redirect_uris":["http://localhost:8080/callback*",``
``"{hostname}/callback*"],``
``"scope":"openid profile email org.cilogon.userinfo",``
``"domain":"{hostname}",``
``"admin_password":"dsdummies",``
``"client_uri":"https:{hostname}",``
``"logo_uri":"https:{hostname}",``
``"application_type":"web",``
``"comment":"Custos super tenant for production"``
}
Open secret in vault, edit 100000 and change supertenant to "true".
Set supertenant to active.
POST https://{host_name}:30079/tenant-management/v1.0.0/status
{
"client_id":"{client id you got in response to last POST request}",
"status":"ACTIVE",
"super_tenant":true,
"updatedBy":"{admin_username}"
}
It should activate the tenant and the output should be :
{
"tenant_id": "10000000",
"status": "ACTIVE"
}
We used Custos Sharing Service to impose fine-grained authorization to protect resources and provide specific permissions for users and groups to access a protected resource.
We referred to Custos documentation for achieving this: (https://cwiki.apache.org/confluence/display/CUSTOS/PEARC21%253ACustos+SDK+Hands-On+Exercises)
- Custos ID (custos_client_id)
- Custos Secret (custos_client_sec)
- Admin Username (admin_user_name)
- Admin Password (admin_password)
import os
import json
import random, string
from custos.clients.user_management_client import UserManagementClient
from custos.clients.group_management_client import GroupManagementClient
from custos.clients.resource_secret_management_client import ResourceSecretManagementClient
from custos.clients.sharing_management_client import SharingManagementClient
from custos.clients.identity_management_client import IdentityManagementClient
from custos.transport.settings import CustosServerClientSettings
import custos.clients.utils.utilities as utl
from google.protobuf.json_format import MessageToJson
try :
# read settings
custos_settings = CustosServerClientSettings(custos_host='custos.scigap.org',
custos_port='31499',
custos_client_id='CLIENT_ID_HERE',
custos_client_sec='CLIENT_SECRET_HERE')
# create custos user management client
user_management_client = UserManagementClient(custos_settings)
# create custos group management client
group_management_client = GroupManagementClient(custos_settings)
# create custos resource secret client
resource_secret_client = ResourceSecretManagementClient(custos_settings)
# create sharing management client
sharing_management_client = SharingManagementClient(custos_settings)
# create identity management client
identity_management_client = IdentityManagementClient(custos_settings)
# obtain base 64 encoded token for tenant
b64_encoded_custos_token = utl.get_token(custos_settings=custos_settings)
print(b64_encoded_custos_token)
created_groups = {}
admin_user_name = "I_WONT_SHARE_MY_USER_NAME"
admin_password = "GUESS_IT_IF_YOU_CAN"
resource_ids = []
print("Successfully configured all custos clients")
except Exception as e:
raise e
print("Custos Id and Secret may wrong "+ str(e))
def verifiy_user(login_user_id,login_user_password):
print("Login user "+ login_user_id)
login_reponse = identity_management_client.token(token=b64_encoded_custos_token, username=login_user_id, password=login_user_password, grant_type='password')
login_reponse = MessageToJson(login_reponse)
print("Login response: ", login_reponse)
response = user_management_client.get_user(token=b64_encoded_custos_token, username=login_user_id)
print(" Updating user profile... ")
user_management_client.update_user_profile(
token=b64_encoded_custos_token,
username=response.username,
email=response.email,
first_name=response.first_name,
last_name=response.last_name)
print(" User "+ login_user_id + " successfully logged in and updated profile")
print("verifiy_user method is defined")
try:
verifiy_user(admin_user_name,admin_password)
print("Successfully verified user")
except Exception as e:
print("verifiy_user is not defined or user may not be created in the teanant"+ str(e))
def register_users(users):
for user in users:
print("Registering user: " + user['username'])
try:
user_management_client.register_user(token=b64_encoded_custos_token,
username=user['username'],
first_name=user['first_name'],
last_name=user['last_name'],
password=user['password'],
email=user['email'],
is_temp_password=False)
user_management_client.enable_user(token=b64_encoded_custos_token, username=user['username'])
except Exception:
print("User may be already exist")
print("register_users method is defined")
users = [
{
'username': 'parth_user',
'first_name': 'Parth_user',
'last_name': '1',
'password': '12345678',
'email': '[email protected]'
},
{
'username': 'umang_user',
'first_name': 'umang_user',
'last_name': '1',
'password': '12345678',
'email': '[email protected]'
},
{
'username': 'prerna_user',
'first_name': 'prerna_user',
'last_name': '1',
'password': '12345678',
'email': '[email protected]'
}
]
try:
register_users(users)
except Exception:
print("please defined method register_users")
def create_groups(groups):
for group in groups:
try:
print("Creating group: " + group['name'])
grResponse = group_management_client.create_group(token=b64_encoded_custos_token,
name=group['name'],
description=group['description'],
owner_id=group['owner_id'])
resp = MessageToJson(grResponse)
print(resp)
respData = json.loads(resp)
print("Created group id of "+ group['name'] + ": " +respData['id'] )
created_groups[respData['name']] = respData['id']
except Exception as e:
print(e)
print("Group may be already created")
print("create_groups method is defined")
groups = [
{
'name': 'Admin',
'description': 'Group for gateway read only admins',
'owner_id': admin_user_name
},
{
'name': 'Read Only Admin',
'description': 'Group for gateway admins',
'owner_id': admin_user_name
},
{
'name': 'Gateway User',
'description': 'Group for gateway users',
'owner_id': admin_user_name
}
]
try :
create_groups(groups)
except Exception as e:
print(e)
print("please defined method create_groups")
def allocate_users_to_groups(user_group_mapping):
for usr_map in user_group_mapping:
try:
group_id = created_groups[usr_map['group_name']]
print("Assigning user " + usr_map['username'] + " to group " + usr_map['group_name'])
val =group_management_client.add_user_to_group(token=b64_encoded_custos_token,
username=usr_map['username'],
group_id=group_id,
membership_type='Member'
)
resp = MessageToJson(val)
print(resp)
except Exception as e:
print(e)
print("User allocation error")
print("allocate_users_to_groups method is defined")
user_group_mapping = [
{
'group_name': 'Admin',
'username': 'parth_user'
},
{
'group_name': 'Admin',
'username': 'umang_user'
},
{
'group_name': 'Admin',
'username': 'prerna_user'
}
]
try:
allocate_users_to_groups(user_group_mapping)
except Exception:
print("please defined method allocate_users_to_groups")
def allocate_child_group_to_parent_group(gr_gr_mapping):
for gr_map in gr_gr_mapping:
try:
child_id = created_groups[gr_map['child_name']]
parent_id = created_groups[gr_map['parent_name']]
print("Assigning child group " + gr_map['child_name'] + " to parent group " + gr_map['parent_name'])
group_management_client.add_child_group(token=b64_encoded_custos_token,
parent_group_id=parent_id,
child_group_id=child_id)
except Exception:
print("Child group allocation error")
print("allocate_child_group_to_parent_group method is defined")
child_gr_parent_gr_mapping = [
{
"child_name": 'Admin',
"parent_name": 'Read Only Admin'
}
]
try:
allocate_child_group_to_parent_group(child_gr_parent_gr_mapping)
except Exception:
print("please defined method allocate_child_group_to_parent_group")
def create_permissions(permissions):
for perm in permissions:
print("Creating permission " + perm['id'])
try:
sharing_management_client.create_permission_type(token=b64_encoded_custos_token,
client_id=custos_settings.CUSTOS_CLIENT_ID,
id=perm['id'],
name=perm['name'],
description=perm['description'])
except Exception:
print("Permission may be already created")
print("create_permissions method is defined")
permissions = [
{
'id': 'READ',
'name': 'READ',
'description': 'Read permission'
},
{
'id': 'WRITE',
'name': 'WRITE',
'description': 'WRITE permission'
}
]
try :
create_permissions(permissions)
except Exception:
print("please defined method create_permissions")
def create_entity_types(entity_types):
for type in entity_types:
print("Creating entity types " + type['id'])
try:
sharing_management_client.create_entity_type(token=b64_encoded_custos_token,
client_id=custos_settings.CUSTOS_CLIENT_ID,
id=type['id'],
name=type['name'],
description=type['description'])
except Exception:
print("Entity type may be already created")
print("create_entity_types method is defined")
entity_types = [
{
'id': 'PROJECT',
'name': 'PROJECT',
'description': 'PROJECT entity type'
},
{
'id': 'EXPERIMENT',
'name': 'EXPERIMENT',
'description': 'EXPERIMENT entity type'
}
]
try :
create_entity_types(entity_types)
except Exception:
print("please defined method create_entity_types")
def register_resources(resources):
for resource in resources:
id = resource['name'].join(random.choice(string.ascii_letters) for x in range(5))
resource_ids.append(id)
resource['id']=id
print("Register resources " + resource['name'] + " generated ID : "+resource['id'] )
sharing_management_client.create_entity(token=b64_encoded_custos_token,
client_id=custos_settings.CUSTOS_CLIENT_ID,
id=resource['id'],
name=resource['name'],
description=resource['description'],
owner_id=resource['user_id'],
type=resource['type'],
parent_id='')
print("register_resources method is defined")
resources = [
{
'name': 'SEAGRD_EXP',
'description': 'Register an experiment',
'user_id': admin_user_name,
'type': 'EXPERIMENT'
}
]
try:
register_resources(resources)
except Exception as e:
print("Please defined method register_resources")
def share_resource_with_group(gr_sharings):
for shr in gr_sharings:
try:
group_id = created_groups[shr['group_name']]
print("Sharing entity " + shr['entity_id'] + " with group " + shr['group_name'] + " with permission " + shr[
'permission_type'])
sharing_management_client.share_entity_with_groups(token=b64_encoded_custos_token,
client_id=custos_settings.CUSTOS_CLIENT_ID,
entity_id=shr['entity_id'],
permission_type=shr['permission_type'],
group_id=group_id)
except Exception as e:
print("Sharing may be already created: "+ str(e))
print("share_resource_with_group method is defined")
gr_sharings = [{
"entity_id": resource_ids[0],
"permission_type": "READ",
"type": "EXPERIMENT",
"group_name": 'Read Only Admin'
}]
try:
share_resource_with_group(gr_sharings)
except Exception as e:
print("please defined method share_resource_with_group")
def share_resource_with_user(sharings):
for shr in sharings:
try:
print("Sharing entity " + shr['entity_id'] + " with user " + shr['user_id'] + " with permission " + shr[
'permission_type'])
sharing_management_client.share_entity_with_users(token=b64_encoded_custos_token,
client_id=custos_settings.CUSTOS_CLIENT_ID,
entity_id=shr['entity_id'],
permission_type=shr['permission_type'],
user_id=shr['user_id']
)
except Exception as e:
print("Sharing error :"+str(e))
print("share_resource_with_user method is defined")
sharings = [
{
"entity_id": resource_ids[0],
"permission_type": "READ",
"type": "EXPERIMENT",
"user_id": "parth_user"
}
]
try :
share_resource_with_user(sharings)
except Exception as e:
print("Please defined method share_resource_with_user")
def check_user_permissions(users):
for user in users:
try:
access = sharing_management_client.user_has_access(token=b64_encoded_custos_token,
client_id=custos_settings.CUSTOS_CLIENT_ID,
entity_id=resource_ids[0],
permission_type="READ",
user_id=user['username'])
usr = user['username']
print("Access for user " + usr + " : " + str(access))
except Exception as e:
print("Permission checking error "+ str(e))
print("check_user_permissions is defined")
try:
check_user_permissions(users)
except Exception as e:
print(e)
print("please defined methos check_user_permissions")
def create_SSH_key(user_id,description):
response = resource_secret_client.add_ssh_credential(token=b64_encoded_custos_token,client_id=custos_settings.CUSTOS_CLIENT_ID,
owner_id=user_id,description=description)
return response.token
def get_SSH_key(token):
return resource_secret_client.get_ssh_credential(token=b64_encoded_custos_token,client_id=custos_settings.CUSTOS_CLIENT_ID,ssh_credential_token=token)
token = create_SSH_key(admin_user_name,'SSH for gateway')
sharing_management_client.create_entity(token=b64_encoded_custos_token,
client_id=custos_settings.CUSTOS_CLIENT_ID,
id=token,
name='SSH Key',
description='SSH for gateway',
owner_id=admin_user_name,
type='SECRET',
parent_id='')
value = get_SSH_key(token)
print(value)
output of this file should look like this:
Registering user: parth
Registering user: umang
Registering user: prerna
Creating group: Admin
Creating group: Read Only Admin
Creating group: Gateway User
Assigning user parth to group Admin
Assigning user umang to group Admin
Assigning user prerna to group Read Only Admin
Assigning child group Admin to parent group Read Only Admin
Creating permission OWNER
Creating permission READ
Creating permission WRITE
Creating entity types PROJECT
Creating entity types EXPERIMENT
Register resources SEAGRD_EXP generated ID : OSEAGRD_EXPNSEAGRD_EXPLSEAGRD_EXPdSEAGRD_EXPR
Sharing entity OSEAGRD_EXPNSEAGRD_EXPLSEAGRD_EXPdSEAGRD_EXPR with user parth with permission READ
Sharing entity OSEAGRD_EXPNSEAGRD_EXPLSEAGRD_EXPdSEAGRD_EXPR with group Read Only Admin with permission READ
Access for user parth : True
Access for user umang : True
Access for user prerna : True
For testing custos we created a RESTFUL flask API that interacts with the deployed tenant. For our analysis we have exposed four endpoints:
Endpoint: /register-user
Request Type: [POST]
Request:
{
"username": "kapil_user1",
"first_name": "kapil_user",
"last_name": "abc",
"password": "12345678",
"email": "[email protected]"
}
Response:
{
"code": "success",
"message": "User Registered!"
}