First attempt at managing the nfcore org with pulumi

[bebosudo]

Hey Edmund, late for the hackaton, but I worked on issue #11 as we discussed.

This is a draft PR to manage the whole org with pulumi. It turns out it's easy enough once you get the feel of it.

I created a test organization test-nfcore and I used it to test the stack.
I'm not sure how to move things forwards now, and how to e.g. store the secret org token in GH to run this automatically once the org structure is changed? Just by using a GHA that sets up the token by taking it from a GH secret every time?
I used this blogpost from pulumi to get started:

pulumi config set github:token "github_pat_***" --secret
pulumi config set github:owner test-nfcore

Which created a Pulumi.dev.yaml file (not commited in this PR). Any idea?

[github-actions]

Terraform Format and Style 🖌success

Terraform Initialization ⚙️success

Terraform Validation 🤖success

Validation Output

Success! The configuration is valid.

Terraform Plan 📖success

Show Plan

terraform

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # github_branch.modules_master will be created
  + resource "github_branch" "modules_master" {
      + branch        = "master"
      + etag          = (known after apply)
      + id            = (known after apply)
      + ref           = (known after apply)
      + repository    = "modules"
      + sha           = (known after apply)
      + source_branch = "main"
      + source_sha    = (known after apply)
    }

  # github_branch_default.modules will be created
  + resource "github_branch_default" "modules" {
      + branch     = "master"
      + id         = (known after apply)
      + rename     = false
      + repository = "modules"
    }

  # github_organization_settings.nf-core will be created
  + resource "github_organization_settings" "nf-core" {
      + advanced_security_enabled_for_new_repositories               = false
      + billing_email                                                = (sensitive value)
      + blog                                                         = "http://nf-co.re"
      + default_repository_permission                                = "read"
      + dependabot_alerts_enabled_for_new_repositories               = false
      + dependabot_security_updates_enabled_for_new_repositories     = false
      + dependency_graph_enabled_for_new_repositories                = false
      + description                                                  = "A community effort to collect a curated set of analysis pipelines built using Nextflow."
      + email                                                        = "[email protected]"
      + has_organization_projects                                    = true
      + has_repository_projects                                      = true
      + id                                                           = (known after apply)
      + members_can_create_internal_repositories                     = false
      + members_can_create_pages                                     = false
      + members_can_create_private_pages                             = false
      + members_can_create_private_repositories                      = false
      + members_can_create_public_pages                              = false
      + members_can_create_public_repositories                       = false
      + members_can_create_repositories                              = false
      + members_can_fork_private_repositories                        = false
      + name                                                         = "nf-core"
      + secret_scanning_enabled_for_new_repositories                 = false
      + secret_scanning_push_protection_enabled_for_new_repositories = false
      + twitter_username                                             = "nf_core"
      + web_commit_signoff_required                                  = false
    }

  # github_repository.modules will be created
  + resource "github_repository" "modules" {
      + allow_auto_merge            = false
      + allow_merge_commit          = false
      + allow_rebase_merge          = false
      + allow_squash_merge          = true
      + allow_update_branch         = true
      + archived                    = false
      + auto_init                   = false
      + default_branch              = (known after apply)
      + delete_branch_on_merge      = false
      + description                 = "Repository to host tool-specific module files for the Nextflow DSL2 community!"
      + etag                        = (known after apply)
      + full_name                   = (known after apply)
      + git_clone_url               = (known after apply)
      + has_discussions             = false
      + has_downloads               = true
      + has_issues                  = true
      + has_projects                = true
      + has_wiki                    = false
      + homepage_url                = "https://nf-co.re"
      + html_url                    = (known after apply)
      + http_clone_url              = (known after apply)
      + id                          = (known after apply)
      + is_template                 = false
      + merge_commit_message        = "PR_TITLE"
      + merge_commit_title          = "MERGE_MESSAGE"
      + name                        = "modules"
      + node_id                     = (known after apply)
      + primary_language            = (known after apply)
      + private                     = (known after apply)
      + repo_id                     = (known after apply)
      + squash_merge_commit_message = "COMMIT_MESSAGES"
      + squash_merge_commit_title   = "PR_TITLE"
      + ssh_clone_url               = (known after apply)
      + svn_url                     = (known after apply)
      + topics                      = [
          + "dsl2",
          + "modules",
          + "nextflow",
          + "nf-core",
          + "nf-test",
          + "pipelines",
          + "workflows",
        ]
      + visibility                  = "public"

      + security_and_analysis {
          + secret_scanning {
              + status = "disabled"
            }
          + secret_scanning_push_protection {
              + status = "disabled"
            }
        }
    }

  # github_repository.pipelines["methylseq"] will be created
  + resource "github_repository" "pipelines" {
      + allow_auto_merge            = false
      + allow_merge_commit          = true
      + allow_rebase_merge          = true
      + allow_squash_merge          = true
      + archived                    = false
      + default_branch              = (known after apply)
      + delete_branch_on_merge      = false
      + etag                        = (known after apply)
      + full_name                   = (known after apply)
      + git_clone_url               = (known after apply)
      + html_url                    = (known after apply)
      + http_clone_url              = (known after apply)
      + id                          = (known after apply)
      + merge_commit_message        = "PR_TITLE"
      + merge_commit_title          = "MERGE_MESSAGE"
      + name                        = "methylseq"
      + node_id                     = (known after apply)
      + primary_language            = (known after apply)
      + private                     = (known after apply)
      + repo_id                     = (known after apply)
      + squash_merge_commit_message = "COMMIT_MESSAGES"
      + squash_merge_commit_title   = "COMMIT_OR_PR_TITLE"
      + ssh_clone_url               = (known after apply)
      + svn_url                     = (known after apply)
      + topics                      = (known after apply)
      + visibility                  = (known after apply)
    }

  # github_repository.pipelines["testpipeline"] will be created
  + resource "github_repository" "pipelines" {
      + allow_auto_merge            = false
      + allow_merge_commit          = true
      + allow_rebase_merge          = true
      + allow_squash_merge          = true
      + archived                    = false
      + default_branch              = (known after apply)
      + delete_branch_on_merge      = false
      + etag                        = (known after apply)
      + full_name                   = (known after apply)
      + git_clone_url               = (known after apply)
      + html_url                    = (known after apply)
      + http_clone_url              = (known after apply)
      + id                          = (known after apply)
      + merge_commit_message        = "PR_TITLE"
      + merge_commit_title          = "MERGE_MESSAGE"
      + name                        = "testpipeline"
      + node_id                     = (known after apply)
      + primary_language            = (known after apply)
      + private                     = (known after apply)
      + repo_id                     = (known after apply)
      + squash_merge_commit_message = "COMMIT_MESSAGES"
      + squash_merge_commit_title   = "COMMIT_OR_PR_TITLE"
      + ssh_clone_url               = (known after apply)
      + svn_url                     = (known after apply)
      + topics                      = (known after apply)
      + visibility                  = (known after apply)
    }

  # github_team.infrastructure will be created
  + resource "github_team" "infrastructure" {
      + create_default_maintainer = false
      + description               = "The best team according to them"
      + etag                      = (known after apply)
      + id                        = (known after apply)
      + members_count             = (known after apply)
      + name                      = "infrastructure"
      + node_id                   = (known after apply)
      + parent_team_read_id       = (known after apply)
      + parent_team_read_slug     = (known after apply)
      + privacy                   = "closed"
      + slug                      = (known after apply)
    }

  # github_team_members.infrastructure will be created
  + resource "github_team_members" "infrastructure" {
      + id      = (known after apply)
      + team_id = (known after apply)

      + members {
          + role     = "maintainer"
          + username = "mashehu"
        }
      + members {
          + role     = "maintainer"
          + username = "mirpedrol"
        }
    }

Plan: 8 to add, 0 to change, 0 to destroy.

Pusher: @bebosudo, Action: pull_request, Working Directory: ``, Workflow: Terraform

[edmundmiller]

Awesome! We also have nf-core-tf that we made for that purpose as well 😆

I think the repo living here is the best pulumi cloud integration, things got weird when I tried to put the repo in the nf-core-tf org.

As for storing the credentials, we can just make a token and store it in pulumi cloud (once they approve our team for open source) then whenever a PR is merged to master it will run is the idea.

Like I said on Gather town I think starting with Teams is a good way for us to safely get started, the worst case is that we accidentally delete a team and have to recreate it, not losing repos.

[edmundmiller]

That all looks perfect to me, did it work?

[bebosudo]

Yep, it worked fine!
You should have received a notification to join one of the teams I created in the test org.

Also private and public repos were created successfully, but we can keep them out of our pulumi stack at the beginning, as you suggested.

Great to hear that we asked Pulumi to sponsor the project! Ping me once that is up and running

I just noticed that we use sub-teams within the nf-core org, e.g. nf-core/infrastructure/ops, so I'll need to make the script iterative.

[bebosudo]

BTW the Personal Access Token I created in the org needed the following permissions:

Organization permissions:

• Read and Write access to members

Repository permissions:

• Read access to metadata
• Read and Write access to administration

Repository Access

• Access to all current and future repositories owned by the organization [we can skip this for now if we decide not to use pulumi for repos]

[bebosudo]

I reworked the script to manage subteams.

I also added explicit dependencies between resources, otherwise sometimes the resource that associates a team with a repo (or a team with a member) was failing because one of the two resources wasn't created yet.

[edmundmiller]

Awesome work!

[edmundmiller]

To clarify, I think we can add a nf-core-test environment for testing purpose to pulumi. The issue was when I tried to use a repo under nf-core-tf/pulumi which pulumi cloud then thought that was our real org, and messed up the auth.

[edmundmiller]

Okay I've gotten this to deploy with Pulumi cloud. Had to just add the secret manually and couldn't get it to work with Pulumi ESC for whatever reason.

Figured we can circle back around to that when we're comfortable using this for main nf-core org.