Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

make ore openstack gc clean up SSH keypairs #3550

Open
dustymabe opened this issue Aug 1, 2023 · 3 comments
Open

make ore openstack gc clean up SSH keypairs #3550

dustymabe opened this issue Aug 1, 2023 · 3 comments
Assignees

Comments

@dustymabe
Copy link
Member

We've been bumping up into quota limits for SSH keypairs in VexxHost (our openstack provider for kola testing) and I didn't quite understand why because the web interface doesn't show me any keypairs so I didn't think any existed. However checking via the CLI I see a lot. So somehow sometimes keypairs aren't getting cleaned up. We could/should make our GC that we run in the pipeline just detect when stale keypairs exist and delete them too. I think we can just filter by keypairs with a name starting with kola- and created before a certain time:

$ openstack keypair show kola-09efd27b-7512-4a51-b795-24bb82eafdf2
+-------------+-------------------------------------------------+
| Field       | Value                                           |
+-------------+-------------------------------------------------+
| created_at  | 2023-08-01T03:21:24.000000                      |
| fingerprint | 1b:de:08:1c:d4:c1:0f:91:9d:e9:8b:d6:f5:e8:1d:26 |
| id          | kola-09efd27b-7512-4a51-b795-24bb82eafdf2       |
| is_deleted  | False                                           |
| name        | kola-09efd27b-7512-4a51-b795-24bb82eafdf2       |
| private_key | None                                            |
| type        | ssh                                             |
| user_id     | c6dffb4547d04cd2bd7ed672c02e2f6d                |
+-------------+-------------------------------------------------+

For now I unblocked things by running:

$ export OS_REGION_NAME="ca-ymq-1"                                    
$ openstack keypair list -f json | jq -r .[].Name | grep kola | xargs openstack keypair delete
marmijo added a commit to marmijo/fedora-coreos-pipeline that referenced this issue Aug 3, 2023
The kola-openstack tests are passing again. Remove the workaround,
but We still need to implement some additional clean up during
garbage collection: coreos/coreos-assembler#3550

This reverts commit 18252d4.
@marmijo
Copy link
Member

marmijo commented Aug 8, 2023

I've been looking at how to implement this and I'm wondering if the kola-openstack job runs garbage collection at all right now? Is it run by default or do we have to specify? I see that in kola-gcp and kola-azure we are running the garbage collection stages manually, but not in kola-openstack. I can open a PR to run GC in openstack if it needs to be specified manually (i'm assuming it does)

@dustymabe
Copy link
Member Author

In my mind I had cross linked this ticket back here when I opened it, but in practice that didn't happen. See

@marmijo
Copy link
Member

marmijo commented Aug 8, 2023

Ah, perfect. I'll work on that.

@marmijo marmijo self-assigned this Aug 8, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants