Migrating to Dynamic Settings

[garnertb]

After a lot of investigation, I am convinced that GitHub Apps cannot be used as an authentication replacement for PATs if organizational settings are configured statically. I believe (based on comments like this: microsoft#216 (comment)) the root cause of the divergence is a focus/prioritization on the implementation used internally at Microsoft that has caused the static configuration logic to fall behind. Since removing PATs is a high priority for the customer, this issue documents the findings/work that will need to be done to support Git App authentication.

Impacts

• Need to re-implement the logic for programmatically generating and deploying org configurations.
• Need to determine the optimal strategy for the number/type of GitHub Apps to include in the deployment.
• When installing the app, GitHub makes a callback to the app's callback url, which errored out on the one install I tried. Do we need this?
• Do we need different apps for prod/dev environments?
• Bootstrap deployed databases with basic configurations.
  • Each row's enittytype field must be set to organizationsetting
  • Each row's entityid field must be set to the organization's id as an integer.
  • Each row's metadata should be set to a json object similar to this:
  • updated is required for the restart logic.

{
  "type": ["public", "private", "internal"], 
  "active": true, 
   "description": "This is a sample org", 
   "updated": "2022-12-06T14:47:41Z",
   "installations": [{
      "appId": 240324, 
      "installationId": "<installation id # for that org>"
    }], 
   "organizationid": 105976989, 
   "organizationname": "asampleorg",
   "features": ["locked", "createReposDirect"]
}

GitHub App Changes/Decisions

• Use a centralized GitHub App that is public to allow all the sub-orgs to install it with minimal hassle.
• Each app installation will have the same appId but a unique installationId, which needs to be added in the installations array of each org.

Environment variable changes

• GitHub App(s) configurations need to be added in all execution contexts.
• App's private key needs to be injected into the GITHUB_APP_*_KEY env vars or accessible as a local file that has it's path set in GITHUB_APP_*_KEY_FILE

export GITHUB_APP_OPERATIONS_APP_ID='240324'
export GITHUB_APP_OPERATIONS_KEY=''
export GITHUB_APP_OPERATIONS_KEY_FILE='../key.pem'
export GITHUB_APP_OPERATIONS_SLUG='github-app-operations'

# Use the same app for all purposes in the app
# Not sure of value of creating an app for each due to installation complexity
export GITHUB_APP_SECURITY_APP_ID=$GITHUB_APP_OPERATIONS_APP_ID
export GITHUB_APP_SECURITY_KEY=$GITHUB_APP_OPERATIONS_KEY
export GITHUB_APP_SECURITY_KEY_FILE=$GITHUB_APP_OPERATIONS_KEY_FILE
export GITHUB_APP_SECURITY_SLUG='github-app-security'

export GITHUB_APP_DATA_APP_ID=$GITHUB_APP_OPERATIONS_APP_ID
export GITHUB_APP_DATA_KEY=$GITHUB_APP_OPERATIONS_KEY
export GITHUB_APP_DATA_KEY_FILE=$GITHUB_APP_OPERATIONS_KEY_FILE
export GITHUB_APP_DATA_SLUG='github-app-data'

export GITHUB_APP_UI_APP_ID=$GITHUB_APP_OPERATIONS_APP_ID
export GITHUB_APP_UI_KEY=$GITHUB_APP_OPERATIONS_KEY
export GITHUB_APP_UI_KEY_FILE=$GITHUB_APP_OPERATIONS_KEY_FILE
export GITHUB_APP_UI_SLUG='github-app-ui'

export GITHUB_APP_ACTIONS_APP_ID=$GITHUB_APP_OPERATIONS_APP_ID
export GITHUB_APP_ACTIONS_KEY=$GITHUB_APP_OPERATIONS_KEY
export GITHUB_APP_ACTIONS_KEY_FILE=$GITHUB_APP_OPERATIONS_KEY_FILE
export GITHUB_APP_ACTIONS_SLUG='github-app-slug'

export GITHUB_APP_JOBS_APP_ID=$GITHUB_APP_OPERATIONS_APP_ID
export GITHUB_APP_JOBS_KEY=$GITHUB_APP_OPERATIONS_KEY
export GITHUB_APP_JOBS_KEY_FILE=$GITHUB_APP_OPERATIONS_KEY_FILE
export GITHUB_APP_JOBS_SLUG='github-app-jobs'

export GITHUB_APP_OPERATIONS_APP_ID=$GITHUB_APP_OPERATIONS_APP_ID
export GITHUB_APP_OPERATIONS_KEY=$GITHUB_APP_OPERATIONS_KEY
export GITHUB_APP_OPERATIONS_KEY_FILE=$GITHUB_APP_OPERATIONS_KEY_FILE
export GITHUB_APP_OPERATIONS_SLUG='github-operations'

export GITHUB_APP_ONBOARDING_APP_ID=$GITHUB_APP_OPERATIONS_APP_ID
export GITHUB_APP_ONBOARDING_KEY=$GITHUB_APP_OPERATIONS_KEY
export GITHUB_APP_ONBOARDING_KEY_FILE=$GITHUB_APP_OPERATIONS_KEY_FILE
export GITHUB_APP_ONBOARDING_SLUG='github-app-onboarding'

export GITHUB_APP_UPDATES_APP_ID=$GITHUB_APP_OPERATIONS_APP_ID
export GITHUB_APP_UPDATES_KEY=$GITHUB_APP_OPERATIONS_KEY
export GITHUB_APP_UPDATES_KEY_FILE=$GITHUB_APP_OPERATIONS_KEY_FILE
export GITHUB_APP_UPDATES_SLUG='github-app-updates'

• Remove these environment variables from all execution contexts

export GITHUB_CENTRAL_OPERATIONS_TOKEN=
export GITHUB_ORGANIZATIONS_FILE=

Related

• Dynamic organization settings example? microsoft/opensource-management-portal#496

How do dynamic settings update?

• The logic responsible for updating dynamic settings is located at business/operations/index.ts.
• By default, dynamic settings are checked every 2 minutes and the app will shutdown if it detects a change.
• Shutdown Logic simply exits with a status code of 0, which I assume delegates restarting to a container management service.

[DevOptimusPrime]

ENV VAR CHANGES

• Remove GITHUB_CENTRAL_OPERATION_TOKEN environment variable
• Add the following new environment variables:

GITHUB_APP_OPERATIONS_ID
GITHUB_APP_OPERATIONS_KEY
GITHUB_APP_OPERATIONS_SLUG
FEATURE_FLAG_ALLOW_USERS_TO_VIEW_LOCKED_ORG_DETAILS

• GITHUB_APP_OPERATIONS_ID is the appId of the Github App being used

• GITHUB_APP_OPERATIONS_KEY is a base64 encoded value of the Github App private key pem

• GITHUB_APP_OPERATIONS_SLUG is a unique identifier

• FEATURE_FLAG_ALLOW_USERS_TO_VIEW_LOCKED_ORG_DETAILS is a boolean toggle (1 or 0) to allow users to land on a custom view when an org is locked for joining

• New env vars should be added to the main portal webapp and the cache-builder and the firehose

DATABASE CHANGES

• Github App installation access token replacement for a user PEM also requires the org environment information to come from the postgres database
• Each org requires a record in the organizationsettings table
  • Each record requires 3 values:
    • The entitytype column is always a value of organizationsetting
    • The entityid column is the Github ID for the organization
    • The metadata column is a JSON blob with the following schema

{
	"name": "ASampleOrg", 
	"type": ["public", "private", "internal"], 
	"active": true, 
	"updated": "2022-12-06T14:47:41Z", 
	"description": "This is a sample org", 
	"installations": [{
		"appId": 240324, 
		"installationId": 29886511
	}],
	"features": ["locked", "createReposDirect"], 
	"organizationid": 105976989, 
	"organizationname": "ASampleOrg"
}

• The easiest way to update the table for testing is to insert a new organization record and then update the JSON blob with the following syntax

Insert new record

INSERT INTO public.organizationsettings
VALUES ('organizationsetting', 12345)

Update the record

UPDATE public.organizationsettings
SET metadata = '{"name":"ASampleOrg","type":["public","private","internal"],"active":true,"updated":"2022-12-06T14:47:41Z","description":"This is a sample org","installations":[{"appId":123,"installationId":890}],"features":["locked", "createReposDirect"]"organizationid":12345,"organizationname":"ASampleOrg"}'
WHERE entityid = '12345'

GITHUB APP INFO

• The new organizationsetting record in the database requires several pieces of information which are available via the Github API
• The most comprehensive source for all of the information required is the Github app installations endpoint which shows an aggregate of all the orgs where an app is installed
• This endpoint requires a JWT token
• This gh cli command piped to jq will output a JSON response containing information for all the organizations an app is installed to
  • Inside each returned JSON object:
    • id is the installationId for the postgres JSON metadata
    • appid is the appId for the postgress JSON metadata
    • account.id is the organizationid for the postgres JSON metadata as well as the entityid column

gh api \
-H "Authorization: Bearer JWTTOKENSTRING" \
-H "Accept: application/vnd.github+json" \
/app/installations | jq

GOTCHAS

• The Github app auth callback can only have a single entry
• Make sure the pem env var is a single base64 encoded
• You will need the Github app pem to generate a jwt to use the gh api route to gather data needed for the organizationsettings table

[github-actions]

This issue has been identified as stale because it has gone 30 days with no activity.
The issue will be closed in 10 days. If this is incorrect, simply comment on the issue, or remove the stale label.