Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add instructions for adding audits table index #58

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add instructions for adding audits table index #58

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
111 changes: 72 additions & 39 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,29 +11,29 @@ Create a new rack locally using your AWS Gov Cloud credentials

You can pass the same [parameters](https://docs.convox.com/installation/production-rack/aws/) as in the UI using the form key=value

$ convox rack install aws gov-rack region=us-gov-east-1
convox rack install aws gov-rack region=us-gov-east-1

don't forget to pass the gov cloud region

## Application Setup

### Clone this repository

$ git clone https://github.com/convox/console-app
git clone https://github.com/convox/console-app

### Create the Console application

You can use any name you like but this document will assume the name `console`.

$ convox apps create console
convox apps create console

### Set up private registry

Convox will provide credentials for a private registry to access the Console images.

Substitute `USERNAME` and `PASSWORD` in this command to add the private registry to your Rack.

$ convox registries add enterprise.convox.com USERNAME PASSWORD
convox registries add enterprise.convox.com USERNAME PASSWORD

## Resource Stack Setup

Expand All @@ -45,25 +45,25 @@ You can use any name you like but the rest of this document will assume the name

You can do this easily via your AWS Web Console, uploading the `formation.json` at the appropriate stage, or using the `aws cli`:

$ aws cloudformation create-stack --stack-name console-resources --capabilities CAPABILITY_IAM --template-body file://formation.json
aws cloudformation create-stack --stack-name console-resources --capabilities CAPABILITY_IAM --template-body file://formation.json

If you are using AWS GovCloud, you have to set the `AwsArn` parameter as `aws-us-gov`:

$ aws cloudformation create-stack --stack-name console-resources --capabilities CAPABILITY_IAM --parameters ParameterKey=AwsArn,ParameterValue=aws-us-gov --template-body file://formation.json
aws cloudformation create-stack --stack-name console-resources --capabilities CAPABILITY_IAM --parameters ParameterKey=AwsArn,ParameterValue=aws-us-gov --template-body file://formation.json

Wait for this stack to fully complete (can take ~10 minutes to complete depending on AWS).

### Configure Console environment

You will need to have [jq](https://stedolan.github.io/jq/) installed if you don't already.

$ bin/export-env console-resources | convox env set -a console
bin/export-env console-resources | convox env set -a console

## License Setup

Convox will provide you a license key for your Console.

$ convox env set -a console LICENSE_KEY=...
convox env set -a console LICENSE_KEY=...

## Custom Domain Setup

Expand All @@ -74,7 +74,7 @@ Decide on a custom domain for your Console. These instructions will assume `cons
Create an SSL certificate for your application. You can use `convox certs import` to load a certificate
that you create manually or you can use the Rack's built-in certificate generator.

$ convox certs generate console.example.org
convox certs generate console.example.org

If you use the automatic certificate generation you will need to accept the certificate validation email that will be sent to the DNS administrator of the domain.

Expand All @@ -84,25 +84,25 @@ Create a CNAME record for this domain to point at the `Router` attribute shown w

### Configure app environment

$ convox env set -a console HOST=console.example.org
convox env set -a console HOST=console.example.org

### (OPTIONAL) Internal Mode

To make the Console only accessible inside the VPC, you will need to set Internal mode.

$ convox env set -a console INTERNAL=true
convox env set -a console INTERNAL=true

(If you are deploying your Console app to an older v2 Rack, this will require your Rack to have the parameter `Internal=Yes` set)

## Deploy the Console

Deploy the application contained in this repository.

$ convox deploy -a console
convox deploy -a console

### Configure Console parameters (only required for older v2 Racks)

$ convox apps params set RackUrl=Yes -a console
convox apps params set RackUrl=Yes -a console

## (OPTIONAL) Integration Setup

Expand All @@ -118,51 +118,86 @@ Use the following callback URL(s) for each service:

Once created, set the appropriate environment variables on your Console application:

$ convox env set -a console GITHUB_CLIENT_ID=... GITHUB_CLIENT_SECRET=...
$ convox env set -a console GITLAB_CLIENT_ID=... GITLAB_CLIENT_SECRET=...
$ convox env set -a console SLACK_CLIENT_ID=... SLACK_CLIENT_SECRET=...
convox env set -a console GITHUB_CLIENT_ID=... GITHUB_CLIENT_SECRET=...
convox env set -a console GITLAB_CLIENT_ID=... GITLAB_CLIENT_SECRET=...
convox env set -a console SLACK_CLIENT_ID=... SLACK_CLIENT_SECRET=...

If you're using GitHub, you'll need to set a random webhook secret:

$ convox env set -a console GITHUB_WEBHOOK_SECRET=...
convox env set -a console GITHUB_WEBHOOK_SECRET=...

If you'd like to use GitHub Enterprise, you'll also need to set the host:

$ convox env set -a console GITHUB_ENTERPRISE_CLIENT_ID=... GITHUB_ENTERPRISE_CLIENT_SECRET=... GITHUB_ENTERPRISE_HOST=github.mycompany.org
convox env set -a console GITHUB_ENTERPRISE_CLIENT_ID=... GITHUB_ENTERPRISE_CLIENT_SECRET=... GITHUB_ENTERPRISE_HOST=github.mycompany.org

Promote the environment changes

$ convox releases promote -a console
convox releases promote -a console

## (OPTIONAL) LDAP Authentication

You can provide credentials for a secure (TLS) LDAP endpoint to use for authentication.

$ convox env set -a console AUTHENTICATION=ldap
$ convox env set -a console LDAP_ADDR=auth.example.org:636 LDAP_BIND=uid=%s,dc=example,dc=org
convox env set -a console AUTHENTICATION=ldap
convox env set -a console LDAP_ADDR=auth.example.org:636 LDAP_BIND=uid=%s,dc=example,dc=org

Set `LDAP_BIND` to a full bind string where `%s` will be substituted for the user's email address.

If your LDAP server does not have a valid certificate issued by a known CA, you can disable certificate validation:

$ convox env set -a console LDAP_VERIFY=no
convox env set -a console LDAP_VERIFY=no

Promote the environment changes

$ convox releases promote -a console
convox releases promote -a console

## (OPTIONAL) SAML Authentication

You can provide configuration details to use SAML for authentication.

$ convox env set -a console AUTHENTICATION=saml
$ convox env set -a console SAML_METADATA=https://login.microsoftonline.com/common/FederationMetadata/2007-06/FederationMetadata.xml
convox env set -a console AUTHENTICATION=saml
convox env set -a console SAML_METADATA=https://login.microsoftonline.com/common/FederationMetadata/2007-06/FederationMetadata.xml

`SAML_METADATA` should be set to the metadata endpoint for your SAML Identity Provider. This varies from provider to provider so please check your documentation from them.

Promote the environment changes

$ convox releases promote -a console
convox releases promote -a console

## (OPTIONAL) Audit logs UI Search

To enable the search feature in the Audit Logs UI it's necessary to create some DynamoDB Indexes and backfill them using a script.
[Indexes.sh](./dynamo/audits/indexes.sh) contains the commands necessary for creating the indexes.
Export AWS credentials in your terminal with the `dynamodb:UpdateTable` permission and the TABLE_PREFIX env variable that you can retrieve using the `bin/export-env` script then run each command separately.

You can temporary attach the following police to an IAM user

```json
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": "dynamodb:UpdateTable",
"Resource": "*"
}
]
}
```

Indexes take a while to create and you can only create one at a time. Check the audits table info in Dynamo UI to verify the index status.

![Dynamo Index UI](./dynamo/index-create.png)

After all indexes are created, run the following command to backfill the indexes

```sh
convox run web -a console dynamo-audit-gsi
Running detached process... OK, someid
```

Confirm that the process is running with `convox ps -a console` and wait for it to complete. Depending on how many records you have in the audits table this process can take a while.

## Steps below are only necessary if you're installing in Gov Cloud

Expand All @@ -174,13 +209,12 @@ Then go to the Account Tab and click "Reset CLI Key" - run the output command in

Now, run the following command

$ convox rack mv gov-rack orgName/gov-rack
convox rack mv gov-rack orgName/gov-rack

Where `gov-rack` is the name of the rack you created and `orgName` the name of the organization you created in the console.

Go to the "Racks" tab and confirm your rack was moved.


### Create an AWS Runtime Integration

Note that this step is only necessary once.
Expand All @@ -193,7 +227,6 @@ Before creating the stack, make sure to check the box near "I acknowledge that A

This stack is creating an IAM Role that the console will use to make changes to rack resources on your behalf.


Wait 1-2 minutes and refresh the Integrations UI to confirm the integration was installed

You can now use this Runtime to create additional racks directly from the console.
Expand All @@ -217,22 +250,22 @@ If you used a custom value for any of the stack parameters, make sure to include
You can check the "Parameters" section of the CloudFormation stack if you're not sure.
E.g: if you have used a custom value in TablePrefix:

```sh
aws cloudformation update-stack --stack-name change-me --capabilities CAPABILITY_IAM --parameters ParameterKey=TablePrefix,UsePreviousValue=true --template-body file://formation.json
```
```sh
aws cloudformation update-stack --stack-name change-me --capabilities CAPABILITY_IAM --parameters ParameterKey=TablePrefix,UsePreviousValue=true --template-body file://formation.json
```

After the stack updates successfully, export the values to your console app:

**Warning**

If running a very old version of the console, it's possible that the `bin/export-env` script will output a `RACK_KEY` and `SESSION_KEY` values different than the ones you already set(you can get the current set values by running `convox env -a console`). If that's the case, just remove the variables from the `bin/export-env` output and set the other ones. You can also rollback the release if you forgot to remove them.

```sh
bin/export-env change-me | convox env set -a console
```
```sh
bin/export-env change-me | convox env set -a console
```

And finally
And finally

```sh
convox deploy -a console
```
```sh
convox deploy -a console
```
56 changes: 56 additions & 0 deletions dynamo/audits/indexes.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,56 @@
# Run each command separateley. The GSIs take some time to finish creating. You can inspect their status in the AWS Dynamo UI
# Export AWS credentials with Dynamo write permissions and also the TABLE_PREFIX env variable
# Create the GSIs before running the backfill binary

aws dynamodb update-table \
--table-name $TABLE_PREFIX-audit-logs \
--attribute-definitions AttributeName=timestamp,AttributeType=S AttributeName=user#org,AttributeType=S \
--global-secondary-index-updates \
"[
{
\"Create\": {
\"IndexName\": \"user-org-index\",
\"KeySchema\": [{\"AttributeName\":\"user#org\",\"KeyType\":\"HASH\"},
{\"AttributeName\":\"timestamp\",\"KeyType\":\"RANGE\"}],
\"Projection\":{
\"ProjectionType\":\"ALL\"
}
}
}
]"

aws dynamodb update-table \
--table-name $TABLE_PREFIX-audit-logs \
--attribute-definitions AttributeName=timestamp,AttributeType=S AttributeName=rack-name#org,AttributeType=S \
--global-secondary-index-updates \
"[
{
\"Create\": {
\"IndexName\": \"rack-name-org-index\",
\"KeySchema\": [{\"AttributeName\":\"rack-name#org\",\"KeyType\":\"HASH\"},
{\"AttributeName\":\"timestamp\",\"KeyType\":\"RANGE\"}],
\"Projection\":{
\"ProjectionType\":\"ALL\"
}
}
}
]"


aws dynamodb update-table \
--table-name $TABLE_PREFIX-audit-logs \
--attribute-definitions AttributeName=timestamp,AttributeType=S AttributeName=user#rack-name#org,AttributeType=S \
--global-secondary-index-updates \
"[
{
\"Create\": {
\"IndexName\": \"user-rack-name-org-index\",
\"KeySchema\": [{\"AttributeName\":\"user#rack-name#org\",\"KeyType\":\"HASH\"},
{\"AttributeName\":\"timestamp\",\"KeyType\":\"RANGE\"}],
\"Projection\":{
\"ProjectionType\":\"ALL\"
}
}
}
]"

Binary file added dynamo/index-create.png
View file
Open in desktop
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.