Fixes #27604 - added Fedora|Redhat CoreOS and IM #8042

lzap · 2020-10-02T11:46:46Z

This patch adds two new OSes:

Fedora CoreOS
RedHat CoreOS

Fedora CoreOS uses quite lengthy version 32.20200907.3.0 which
unfortunately cannot be stored into major or minor version field which
is by default validated to be numeric. Thing is, after validation both
major and minor are converted to strings and stored in database as
strings. Operating system versions are never compared as numbers (they
are fetched back as strings), therefore those numric validations were
removed - there is no reason to enforce them to be numbers.

We are not doing a good job explaining to our users what exactly should
be entered into major, minor and release_name fields. As part of the
patch, three new *_help methods are added to the operating system
where a user-facing help string can be displayed for major, minor and
release_name fields with closer explanation.

Warning: This is not end-to-end FedoraCoreOS provisioning, but it's probably the first in the series to enable the workflow.

test/models/operatingsystems/operatingsystems_test.rb

db/seeds.d/100-installation_media.rb

app/models/operatingsystems/coreos.rb

theforeman-bot · 2020-10-02T11:47:08Z

Issues: #27604

db/seeds.d/100-installation_media.rb

houndci-bot · 2020-10-02T11:52:15Z

db/seeds.d/100-installation_media.rb

+    {
+      :name => "CentOS 7 mirror",
+      :os_family => "Redhat",
+      :path => "http://mirror.centos.org/centos/$major/os/$arch"


Style/TrailingCommaInHashLiteral: Put a comma after the last item of a multiline hash.

This is not resolved. We are enforcing comma here.

Elyytscha · 2020-10-05T09:05:51Z

Hi,

just wanted to let you know that i'm testing this patch actually

our ci builds actually our foreman container image in version 2.1.3 on top this pull request gets patched into the image (slighty modificated [commented some apipie stuff out because foreman 2.1.3 does not seem to support it so it breaks while it starts]

but i need still some time and on the weekend i mostly enjoy my weekend so mostly there i do no computer relevant stuff :)

best regards

Elyytscha · 2020-10-05T10:33:19Z

the deployment on our testsystem, was successfull

(maybe to mention our testsystem provisions vm's for our testing, ci and development zones so its basically also a prodsystem)

i did no heavy testing yet, but i can say our foreman and foremanproxy are using the same patched code

everything what worked before this patch is still operational, so i can still provision vm's the same way we did before

also the options for RHCOS // FHCOS are looking good for me, also the descriptions it does look like your screenshots

but when i try now to provision a rhcos vm witih this patch i get the following:

maybe i missed a file to patch? but i seached through my patched filed and the pullrequest and wasn't able to find this function

edit : the error as text:

Unable to render PXELinux template 'RHCOS PXE': undefined method `rhcos_attributes' for #<Foreman::Renderer::Scope::Provisioning:0x00007febc04bf060> Did you mean? coreos_attributes

edit2: this is the used pxe template:

DEFAULT pxeboot
TIMEOUT 20
PROMPT 0
LABEL pxeboot
    KERNEL <%= @kernel %>
    APPEND ip=dhcp rd.neednet=1 initrd=<%= @initrd %> coreos.autologin=yes coreos.first_boot=1 coreos.inst=yes coreos.inst.callhome_url=<%= foreman_url('provision')%> coreos.inst.install_dev=sda coreos.inst.image_url=http://foreman.domain.com:8089/pxeboot/rhcos/rhcos-4.5.6-x86_64-metal.x86_64.raw.gz coreos.inst.ignition_url=http://foreman.domain.com:8089/pxeboot/rhcos/<%= @host.params['role'] %>.ign

what i can confirm is that the initramfs and the kernel for rhcos was downloaded successfully with this to the tftp boot folder:

ekohl · 2020-10-07T08:49:47Z

For completeness, this came from https://community.theforeman.org/t/rhcos-fhcos-provisioned-for-openshift/20704. https://community.theforeman.org/t/rhcos-installation-with-foreman/18702 is another user who asked for this support.

lzap · 2020-10-07T13:32:03Z

I've fixed the rendering problem and tests. The diff is small:

diff --git a/lib/foreman/renderer/scope/variables/base.rb b/lib/foreman/renderer/scope/variables/base.rb
index 25e7ab9e9..c2342f975 100644
--- a/lib/foreman/renderer/scope/variables/base.rb
+++ b/lib/foreman/renderer/scope/variables/base.rb
@@ -36,7 +36,7 @@ module Foreman
             @template_url = params['url']
           end
 
-          %w(coreos aif memdisk ZTP).each do |name|
+          %w(coreos fcos rhcos aif memdisk ZTP).each do |name|
             define_method("#{name}_attributes") do
               @mediapath = mediumpath(@medium_provider) if medium
             end

That will fix it.

Elyytscha

i tested it with this change now i was able to create a rhcos vm in foreman but it was not able to find its kernel

Elyytscha · 2020-10-07T14:48:31Z

app/models/operatingsystems/rhcos.rb

+  # http://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.5/4.5.6/rhcos-installer-initramfs.x86_64.img
+  #
+  PXEFILES = {
+    kernel: 'rhcos-installer-kernel-$arch',


do i have to set the arch in a special way?

i was able to create a rhcos vm in foreman but it hangs at finding the kernel:

my OS does look like this:

or is it possible that the var here is not working correctly?

It's a bug! I am rebasing this PR to fix this, good catch. Here is the diff:

diff --git a/app/models/operatingsystems/fcos.rb b/app/models/operatingsystems/fcos.rb index 1d894c54d..8286f1b34 100644 --- a/app/models/operatingsystems/fcos.rb +++ b/app/models/operatingsystems/fcos.rb @@ -13,6 +13,10 @@ class Fcos < Operatingsystem 'fcos' end + def bootfile(medium_provider, type) + medium_provider.interpolate_vars(super).to_s + end + def pxedir(medium_provider = nil) medium_provider.interpolate_vars('prod/streams/$release/builds/$major.$minor/$arch').to_s end diff --git a/app/models/operatingsystems/rhcos.rb b/app/models/operatingsystems/rhcos.rb index 7a2ceefaf..7fa7c7aab 100644 --- a/app/models/operatingsystems/rhcos.rb +++ b/app/models/operatingsystems/rhcos.rb @@ -13,6 +13,10 @@ class Rhcos < Operatingsystem 'rhcos' end + def bootfile(medium_provider, type) + medium_provider.interpolate_vars(super).to_s + end + def pxedir(medium_provider = nil) medium_provider.interpolate_vars('pub/openshift-v$major/$arch/dependencies/rhcos/$major.$minor/$major.$minor.$release').to_s end

Thanks for testing!

Elyytscha · 2020-10-13T14:52:18Z

i tested it for rhcos 4.5.6 and it is working now as far as i can see :)

I will test it for fcos too and then I report the result here in the comment as edit

lzap · 2020-10-14T11:02:35Z

Great, next up is adding the 3rd boot file for download (the image). But let's wait until this is merged.

lzap · 2020-10-14T11:04:13Z

@ezr-ondrej I see you assigned yourself, if you don't mind reviewing all three PRs I am planning in this regard. If @Elyytscha can help with testing that would be fantastic! The other one is #8059 and I am planning one more once these are merged. Also quite small.

Elyytscha

https://builds.coreos.fedoraproject.org
instead of

http://builds.coreos.fedoraproject.org

and also the fcos mirror will work:

i noticed this:

the icon problem seems to be existent for rhcos/fcos the url it wants to get the icon from is:

/images/icons16x16/rhcos.png
/images/icons16x16/fcos.png

maybe could be changed to the normal rhel // fedora icons?

Elyytscha · 2020-10-15T18:04:37Z

test/models/operatingsystems/operatingsystems_test.rb

+      'major' => '32',
+      'minor' => '20200907.3.0',
+      'release_name' => 'stable',
+      'kernel' => 'http://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20200907.3.0/x86_64/fedora-coreos-32.20200907.3.0-live-kernel-x86_64',


should be also https

Elyytscha · 2020-10-15T18:04:58Z

test/models/operatingsystems/operatingsystems_test.rb

+      'minor' => '20200907.3.0',
+      'release_name' => 'stable',
+      'kernel' => 'http://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20200907.3.0/x86_64/fedora-coreos-32.20200907.3.0-live-kernel-x86_64',
+      'initrd' => 'http://builds.coreos.fedoraproject.org/prod/streams/stable/builds/32.20200907.3.0/x86_64/fedora-coreos-32.20200907.3.0-live-kernel-x86_64',


should be also https

Elyytscha · 2020-10-15T18:05:19Z

db/seeds.d/100-installation_media.rb

+    {
+      :name => "Fedora CoreOS mirror",
+      :os_family => "Fcos",
+      :path => "http://builds.coreos.fedoraproject.org",


should be https instead, with https i got it working, with http not

test/factories/medium.rb

Elyytscha · 2020-11-12T12:21:21Z

app/models/operatingsystems/rhcos.rb

+  # http://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.5/4.5.6/rhcos-installer-initramfs.x86_64.img
+  #
+  PXEFILES = {
+    kernel: 'rhcos-installer-kernel-$arch',


broken with openshift 4.6.. because of a rename..

https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.6/4.6.1/rhcos-live-initramfs.x86_64.img

i think its good that it was not merged yet!

lzap · 2020-11-18T10:30:36Z

Fixed https, icons and also renamed installer to live for 4.6 release. Thanks for testing! @ezr-ondrej

lzap · 2021-01-20T11:29:18Z

Retriggering tests, 3 failed.

Elyytscha · 2021-05-07T17:27:51Z

for sure i can, its just not perfect and not usable for a general fcos/rhcos provisioning i think, but maybe it helps

<%#
name: RHCOS PXE
%>

DEFAULT pxeboot
TIMEOUT 20
PROMPT 0
LABEL pxeboot
    KERNEL <%= @kernel %>
    APPEND initrd=<%= @initrd %> ignition.platform.id=vmware coreos.inst.insecure=true console=tty0 console=ttyS0 coreos.live.rootfs_url=http://foreman.example.com/pxeboot/rhcos/rhcos-live-rootfs-<%= @host.params['version'] %>.x86_64.img <% @host.managed_interfaces.each do |interface| -%> <% next if !interface.managed? || interface.primary? -%> ip=<%= interface.ip %>::<%= interface.subnet.gateway %>:<%= interface.subnet.mask %>:<%= @host.name %>:ens224:none <% end -%> ip=<%= @host.ip %>::<%= @host.subnet.gateway %>:<%= @host.subnet.mask %>:<%= @host.name %>:ens192:none rd.neednet=1 nameserver=<%= @host.subnet.dns_primary %> nameserver=<%= @host.subnet.dns_secondary %> coreos.autologin=yes coreos.first_boot=1 coreos.inst=yes coreos.inst.callhome_url=<%= foreman_url('provision')%> coreos.inst.install_dev=sda coreos.inst.image_url=http://foreman.example.com/pxeboot/rhcos/rhcos-<%= @host.params['version'] %>-x86_64-metal.x86_64.raw.gz coreos.inst.ignition_url=http://foreman.example.com/pxeboot/rhcos/<%= @host.params['clustername'] %>/<%= @host.params['role'] %>.ign

ignition.platform.id=vmware
dont know how to generalize this.. maybe platform (bare)metal/none for everything?

coreos.live.rootfs_url=http://foreman.example.com/pxeboot/rhcos/rhcos-live-rootfs-<%= @host.params['version'] %>.x86_64.img
this supplies the rhcos live rootfs (which was downloaded in a first step by our openshift4 ansible deployment into the foreman http server, so a vm/baremetal host can access it from there. Also our ansible creates the VM's on foreman via hammer cli, due to this our ansible knows that it has to set the host parameter 'version' to something like 4.6.8, so the vm knows which rhcos/openshift version it should pick up

<% @host.managed_interfaces.each do |interface| -%> <% next if !interface.managed? || interface.primary? -%> ip=<%= interface.ip %>::<%= interface.subnet.gateway %>:<%= interface.subnet.mask %>:<%= @host.name %>:ens224:none <% end -%> ip=<%= @host.ip %>::<%= @host.subnet.gateway %>:<%= @host.subnet.mask %>:<%= @host.name %>:ens192:none
this does our two network interfaces, one for the normal network, the second for the storage network, (a possible pitfall is, the network without the default gateway (storage net) has to be the first in the list, why this, i forgot, but i could look through our redhat tickets if someone wants to know why. The Interface identifiers are hardcoded, could come also from foreman, but we don't set those.

coreos.inst.callhome_url=<%= foreman_url('provision')%>
actually this is without functionality, a relation could be found over here: coreos/coreos-installer#21

coreos.inst.install_dev=sda
again a possible pitfall, I searched ~2 hours to find out that our baremetal server had no sda, they started with sdb, but our vm's are all starting with sda

coreos.inst.image_url=http://foreman.example.com/pxeboot/rhcos/rhcos-<%= @host.params['version'] %>-x86_64-metal.x86_64.raw.gz
same like the rootfs, just for the metal image

coreos.inst.ignition_url=http://foreman.example.com/pxeboot/rhcos/<%= @host.params['clustername'] %>/<%= @host.params['role'] %>.ign
yeah, finally our ansible also created an openshift install via the openshift installer and uploaded boostrap/master/worker.ign into the foreman http server. When our ansible then creates the vm's it sets a host param role=boostrap/master/worker, so the vm knows which cluster component it is, to dont get mixed up with our different clusters, we also have the host param clustername, so the *.ign files can reside in a dir named same like the openshift cluster

ah, and i forgot, the last ugly thing is our openshift ansible has to cancel the foreman host build after the pxe boot happend, because i never found out how to do a working provisioning callback to foreman after the openshift host was installed

why we did it this way: when you want to install openshift via the general UPI method, you will notice that you need a tftp, http server, foreman had all this already up and working so we just used foremans infrastructure for this and integrated the hosts as much as possible into our foreman

i hope this makes sense somehow and i hope it will help a little bit

lzap · 2021-05-14T12:46:02Z

I amended test fixes both for core and katello.

@Elyytscha can you take your last comment and convert it into a PR? We will appreciate the contribution, let's discuss the details there. Parametrization is wanted, but you do not need to parametrize everything - just enough so others can use it too.

ezr-ondrej · 2021-05-14T12:49:24Z

Parametrization is wanted, but you do not need to parametrize everything

I believe we would need to download the rootfs ourselves with the pxe files and change the initrd to include it, instead host params for the version, but apart of that I think it would be nice contribution already.

ezr-ondrej · 2021-05-14T13:00:42Z

[test katello]

ezr-ondrej · 2021-05-14T14:22:17Z

[test katello] not sure what's wrong with katello

lzap · 2021-05-17T06:57:06Z

Aaron Patterson released new version of Psych YAML parser 4.0 which is a major change - by default it safe-loads all YAML. It's good for all of us: ruby/psych#487 anyway Katello codebase has an open ended depednency and that's why it fails:

Psych::DisallowedClass: Tried to load unspecified class: Symbol
/usr/local/rvm/gems/ruby-2.5.1@test_develop_pr_katello-1/gems/psych-4.0.0/lib/psych/class_loader.rb:99:in `find'
/usr/local/rvm/gems/ruby-2.5.1@test_develop_pr_katello-1/gems/psych-4.0.0/lib/psych/class_loader.rb:28:in `load'
(eval):2:in `symbol'
/usr/local/rvm/gems/ruby-2.5.1@test_develop_pr_katello-1/gems/psych-4.0.0/lib/psych/class_loader.rb:32:in `symbolize'
/usr/local/rvm/gems/ruby-2.5.1@test_develop_pr_katello-1/gems/psych-4.0.0/lib/psych/class_loader.rb:84:in `symbolize'
/usr/local/rvm/gems/ruby-2.5.1@test_develop_pr_katello-1/gems/psych-4.0.0/lib/psych/scalar_scanner.rb:74:in `tokenize'
/usr/local/rvm/gems/ruby-2.5.1@test_develop_pr_katello-1/gems/psych-4.0.0/lib/psych/visitors/to_ruby.rb:65:in `deserialize'
/usr/local/rvm/gems/ruby-2.5.1@test_develop_pr_katello-1/gems/psych-4.0.0/lib/psych/visitors/to_ruby.rb:128:in `visit_Psych_Nodes_Scalar'
/usr/local/rvm/gems/ruby-2.5.1@test_develop_pr_katello-1/gems/psych-4.0.0/lib/psych/visitors/visitor.rb:30:in `visit'

I am writing a PSA to the community and I will file a PR for Katello team to unblock the CI: https://community.theforeman.org/t/psa-rubygem-psych-4-0-is-out-and-its-a-breaking-change/23604

lzap · 2021-05-17T07:05:04Z

Here is a patch for Katello plugin: Katello/katello#9365

lzap · 2021-05-18T13:02:24Z

Okay solved the problem with psych 4.0 YAML parser in Katello, rebasing.

ezr-ondrej · 2021-05-18T13:39:21Z

It still doesn't work... different reason now though.

ezr-ondrej · 2021-05-19T14:49:33Z

[test katello]

ezr-ondrej · 2021-05-19T15:12:16Z

[test katello] it is still pulling in psych 4.0 :(

lzap · 2021-05-20T11:20:23Z

[test katello] one more time

to a random reader: if you are reading this and considering getting into software engineering for having fun, lemme tell you: ITS NOT FUN!

ezr-ondrej · 2021-05-20T13:45:59Z

[test katello]

ezr-ondrej · 2021-05-20T14:22:43Z

[test katello]

ezr-ondrej

Nice addition! Thanks @lzap ! 👍
I'd love to see follow-ups so we can provision CoreOSes, so I hope you won't follow on your own advice and become carpenter xD

lzap · 2021-05-21T05:58:38Z

Thanks, @Elyytscha can you submit template changes? What else is left to get this working then? I would love to demo this if we got it working.

Elyytscha · 2021-05-29T08:13:56Z

@lzap i think whats needed now is:

decision how to download the rootfs (coreos.live.rootfs_url=http://foreman.example.com/pxeboot/rhcos/rhcos-live-rootfs-<%= @host.params['version'] %>.x86_64.img) and the metal image into foremans http server (coreos.inst.image_url=http://foreman.example.com/pxeboot/rhcos/rhcos-<%= @host.params['version'] %>-x86_64-metal.x86_64.raw.gz) and code for this
method for provisioning callback, so foreman knows when a node has successfully installed (for now a manual build cancel is needed after the host pxe booted successfully
generic pxe template for rhcos (i will try to create a pullrequest, but im really lacking free time at the moment, i bought a house which im renovating in my freetime and i have to work 40 hours every week)
maybe a new foreman template group (e.g openshift install template) which takes an openshift install.yaml as input and is able to run this https://mirror.openshift.com/pub/openshift-v4/clients/ocp/4.7.13/openshift-install-linux.tar.gz binary to produce openshift install files (master,bootstrap,worker.ign) which are then put into the foreman http server
a positive side effect of this would be we would not need those uncool hostvariables which i used in the pxe template like openshift version and cluster name. the only complicated thing would be to detect/define somehow which host will be a master/worker/boostrap node
all the same for fedora coreos and openshift origin (its basically exact the same, just fedora instead of redhat coreos and other openshift install binarys)

with this 4 points, foreman would be able to install openshift container platform and openshift origin completly native

i wrote you an email so you can reach me faster for details if you need help or if i need help with the pullrequest for the pxe template.

lzap · 2021-05-31T09:21:43Z

Thanks for the writeup. I am currently busy with other things so this is not on my short-term agenda, feel free to file PRs. I did not receive any email tho, its lzap_at_redhat_com if you meant me. I am here to help with reviews.

sasskinn12 · 2021-11-30T08:09:53Z

Hi guys, trying implement something similar as @Elyytscha. Could you suggest or reccomend how to do "step 2" after FCOS successful install. I would like to have it dynamically, not manual cancel.
Eg.

method for provisioning callback, so foreman knows when a node has successfully installed (for now a manual build cancel is needed after the host pxe booted successfully.

Thanks in advice. :)

houndci-bot reviewed Oct 2, 2020

View reviewed changes

test/models/operatingsystems/operatingsystems_test.rb Outdated Show resolved Hide resolved

db/seeds.d/100-installation_media.rb Outdated Show resolved Hide resolved

app/models/operatingsystems/coreos.rb Outdated Show resolved Hide resolved

theforeman-bot added the Legacy JS PRs making changes in the legacy Javascript stack label Oct 2, 2020

theforeman-bot added Needs testing Not yet reviewed labels Oct 2, 2020

lzap force-pushed the fcos-os-27604 branch from a877844 to 867dc92 Compare October 2, 2020 11:52

houndci-bot reviewed Oct 2, 2020

View reviewed changes

lzap force-pushed the fcos-os-27604 branch from 867dc92 to 9adb9a9 Compare October 2, 2020 12:12

ezr-ondrej self-assigned this Oct 7, 2020

lzap force-pushed the fcos-os-27604 branch from 9adb9a9 to ab89e65 Compare October 7, 2020 13:32

Elyytscha reviewed Oct 7, 2020

View reviewed changes

lzap mentioned this pull request Oct 8, 2020

Fixes #31027 - transpiler macros for CoreOS family #8059

Merged

lzap force-pushed the fcos-os-27604 branch from ab89e65 to 28acca7 Compare October 12, 2020 13:55

Elyytscha suggested changes Oct 15, 2020

View reviewed changes

theforeman-bot added Waiting on contributor and removed Not yet reviewed labels Oct 15, 2020

Elyytscha suggested changes Nov 12, 2020

View reviewed changes

lzap force-pushed the fcos-os-27604 branch from 28acca7 to e9e649a Compare November 18, 2020 10:30

theforeman-bot added Needs re-review and removed Waiting on contributor labels Nov 18, 2020

lzap force-pushed the fcos-os-27604 branch from e9e649a to 41bc8ac Compare January 20, 2021 11:28

theforeman-bot added the Waiting on contributor label May 7, 2021

lzap force-pushed the fcos-os-27604 branch from b1f7348 to 17a61d7 Compare May 14, 2021 12:44

theforeman-bot added Needs re-review and removed Waiting on contributor labels May 14, 2021

Fixes #27604 - added Fedora|Redhat CoreOS and IM

113425a

lzap force-pushed the fcos-os-27604 branch from 17a61d7 to 113425a Compare May 18, 2021 13:02

ezr-ondrej approved these changes May 20, 2021

View reviewed changes

theforeman-bot removed the Needs re-review label May 20, 2021

ezr-ondrej merged commit 055489b into theforeman:develop May 20, 2021

lzap deleted the fcos-os-27604 branch May 21, 2021 05:57

lzap added Demo worthy and removed Needs testing labels May 21, 2021

Fixes #27604 - added Fedora|Redhat CoreOS and IM #8042

Fixes #27604 - added Fedora|Redhat CoreOS and IM #8042

Conversation

lzap commented Oct 2, 2020 • edited Loading

theforeman-bot commented Oct 2, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Elyytscha commented Oct 5, 2020 • edited Loading

Elyytscha commented Oct 5, 2020 • edited Loading

ekohl commented Oct 7, 2020

lzap commented Oct 7, 2020

Elyytscha left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Elyytscha commented Oct 13, 2020

lzap commented Oct 14, 2020

lzap commented Oct 14, 2020

Elyytscha left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lzap commented Nov 18, 2020

lzap commented Jan 20, 2021

Elyytscha commented May 7, 2021 • edited Loading

lzap commented May 14, 2021

ezr-ondrej commented May 14, 2021

ezr-ondrej commented May 14, 2021

ezr-ondrej commented May 14, 2021

lzap commented May 17, 2021 • edited Loading

lzap commented May 17, 2021

lzap commented May 18, 2021

ezr-ondrej commented May 18, 2021

ezr-ondrej commented May 19, 2021

ezr-ondrej commented May 19, 2021

lzap commented May 20, 2021

ezr-ondrej commented May 20, 2021

ezr-ondrej commented May 20, 2021

ezr-ondrej left a comment

Choose a reason for hiding this comment

lzap commented May 21, 2021

Elyytscha commented May 29, 2021 • edited Loading

lzap commented May 31, 2021

sasskinn12 commented Nov 30, 2021 • edited Loading

lzap commented Oct 2, 2020 •

edited

Loading

Elyytscha commented Oct 5, 2020 •

edited

Loading

Elyytscha commented Oct 5, 2020 •

edited

Loading

Elyytscha left a comment •

edited

Loading

Elyytscha commented May 7, 2021 •

edited

Loading

lzap commented May 17, 2021 •

edited

Loading

Elyytscha commented May 29, 2021 •

edited

Loading

sasskinn12 commented Nov 30, 2021 •

edited

Loading