Skip to content

Commit

Permalink
Merge branch 'main' into NTRNL-358-update-description-validation-to-h…
Browse files Browse the repository at this point in the history 
…andle-case-where-it-is-required-for-repo-request-and-team-request
  • Loading branch information
@mel-am
mel-am authored Feb 22, 2024
2 parents 3253317 + f533a15 commit 05d6795
Show file tree
Hide file tree
Showing 22 changed files with 480 additions and 110 deletions.
123 changes: 69 additions & 54 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -1,81 +1,96 @@
# GitHub requests app
# GitHub Requests App

![Static Badge](https://img.shields.io/badge/test_coverage-%E2%89%A595%25-green)

The GitHub request application is a tool designed to streamline and automate the process of managing and tracking request for GitHub. This includes the adding, removal or editing of members. Further documentation can be found [here](./docs/).

## Overview

The GitHub request application is a tool designed to streamline and automate the process of managing and tracking request for GitHub. This includes the adding, removal or editing of members.
Each page or user interface, defined by an endpoint, is divided into three components (MVC) and as a best practice the names for the model, view and controller have, when possible, the same start name of the endpoints (e.g. for the `/confirmation` page we have the: `confirmation.controller.ts` and `confirmation.html` files. If models were present, we would have `confirmation.model.ts`)
The GitHub Requests Application is a Node.js-based web application that provides a simple interface. Internal users (members of the Cognito user pool) fill in forms with requested details. When a request is submitted, an issue is created on the dedicated terraform repository, and an email is sent to the user from our `github-request.idp` email address. The email includes a message containing the user's filled-in information.

The issue is then reviewed by the team, and further comments may be requested if necessary. Approval must be granted by two members of the team to avoid misconfigurations.

## Frontend Technologies and Utils

- [NodeJS](https://nodejs.org/)
- [ExpressJS](https://expressjs.com/)
- [Typescript](https://www.typescriptlang.org/)
- [NunJucks](https://mozilla.github.io/nunjucks)
- [GOV.UK Design System](https://design-system.service.gov.uk/)
- [Jest](https://jestjs.io)
- [SuperTest](https://www.npmjs.com/package/supertest)
- [Docker](https://www.docker.com/)
- [Git](https://git-scm.com/downloads)
- [Terraform](https://www.terraform.io/)
- [AWS](https://aws.amazon.com/)

### Config variables

Key | Description | Example Value
----------------|--------------------------- |-------------------------
PATH_SSL_PRIVATE_KEY | Path to ssl private key | `./infrastructure/host/test.key`
PATH_SSL_CERTIFICATE | Path to ssl certificate | `./infrastructure/host/test.cert`
BASE_URL | Base application URL | `http://localhost:3000` (dev mode)
NODE_ENV | Node environment | `development` (or `production`)
PORT | Server port number | `3000`
CDN_HOST | CDN host | `cdn domain`
USER_POOL_ID | ID of the user pool in Amazon Cognito | `secret`
USER_POOL_CLIENT_ID | Client ID of an app registered with the user pool in Amazon Cognito | `secret`
AUTH_SIGN_IN_URL | Authentication sign in URL | `https://cola.service.cabinetoffice.gov.uk/v2/<YOUR_SERVICE>/login`
COOKIE_ID_NAME | The name of the cookie | `github-requests`
COOKIE_PARSER_SECRET | Secret used in validating/calculating the cookie signature | `secret`
COOKIE_SESSION_SECRET | Secret key for signing the session cookie | `secret`
LANDING_PAGE_URL | Github Requests landing Page | `/home/`
LOG_LEVEL | Logging levels | `/home/`
HUMAN | Formatting messages form (default JSON) | `true` (Enable human formatting for log messages)

## Launching the web-app

### Prerequisites

1. Install [NodeJS V20.8](https://nodejs.org/en)
2. Install [Docker](https://www.docker.com/get-started)

### The View
### Running local development environment with Docker

We use `Nunjucks` and `GDS` style/components.
Docker is used to run the application in **development** mode, with tooling setup to detect changes in local `src` directory and reload the container's node server. Ensure that `NODE_ENV=development` is set in the `.env` file.

### The controller
### Building the Docker Image

Generally only POST and GET http methods are allowed, and therefore we will have mainly just the get and post controllers, and it is literally the last successful middleware of the chain that has the duty to respond to the user.
In the `get` method we fetch possible data and pass it to the view/template to be visualized to the user and in the `post` method we save the user data everytime that a page is submitted
1. Create a copy of the `.env.example` file and name it `.env`:
Then run:

## Files Structure
```sh
make docker-build
make docker-up
```

Directory Path | Description
--- | ---
`./.github` | Github folder, includes `PULL_REQUEST_TEMPLATE.md` on how to make a pull request to the project and `dependabot.yml` configuration options for dependency updates.
`./.husky` | Add pre check script, includes `pre-commit` and `pre-push` checks
`./src` | Contains all Typescript code
`./src/app.ts` | Application entry point
`./src/bin/www.ts` | Server configuration
`./src/config/index.ts` | Contains all the application's configurations
`./src/controller` | Business logic and handlers
`./src/middleware` | Middleware functions (Authentication, validation ...)
`./src/model` | OE Session and View Data Model
`./src/routes` | Paths and routes controller (Only GET and POST enabled)
`./src/service` | Interface to the API through SDK
`./src/utils` | Facade for CO services (e.g. logging) and other application utils (navigation, application data ...)
`./src/validation` | Sets of express validator middlewares for each page
`./test` | Jest Test files (`*.spec.ts`, `setup.ts`, and `*.mocks.ts`)
`./view` | Contains all the html nunjucks structure files
`./docs` | Contains documentation files
Others files | Other files related to modules dependency, CI/CD, *git, dockerization, lint, test/typescript configs …
This will then download the necessary dependencies, build the Docker image, and start the application. You will be able to access it on [localhost:3000](localhost:3000).

## ESlint

We use ESlint as both a formatter and code quality assurance. Eslint can also be setup to format on save using a VScode extension:

1. Install the [ESlint VScode extenstion](https://marketplace.visualstudio.com/items?itemName=dbaeumer.vscode-eslint).

2. Open your user settings (JSON) inside VScode and add the following:
```

```js
"editor.formatOnSave": true,
"editor.codeActionsOnSave": { "source.fixAll.eslint": true }
```
```

3. Reload VScode.

## Running local development environment with Docker

Docker is used run the application in development mode, with tooling setup to detect changes in local `src` directory and reload the container's node server.

Follow the steps in [Launching-the-web-app](#Launching-the-web-app), and ensure that `NODE_ENV=development` is set in the `.env` file.

## Launching the web-app

### Prerequisites

1. Install [NodeJS V20.8](https://nodejs.org/en)

2. Install [Docker](https://www.docker.com/get-started)

### Building the Docker Image

1. Create a copy of the ``.env.example`` file and name it `.env`:

Then run:

make docker-build
## Recommendations

make docker-up
1. Use the [Visual Studio Code](https://code.visualstudio.com/) IDE for development.
2. Use the preformatted `PULL_REQUEST_TEMPLATE` by adding meaningful description
3. Make sure test coverage is above `95%`
4. Do not disable husky pre checks locally
5. Use MVC pattern when adding a new page/endpoint, including validation and authentication. Example can be found on the following doc description [here](./docs/Project%20Structure%20and%20Code%20Style.md)
6. **Happy coding**

This will then download the necessary dependencies, build the Docker image, and start the application.
You will be able to access it on [localhost:3000](localhost:3000).
## License

This code is open source software licensed under the [MIT License]("https://opensource.org/licenses/MIT").
91 changes: 91 additions & 0 deletions docs/GitHub Requests (Application & Terraform).md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,91 @@
# GitHub Requests (Application & Terraform)

**GitHub Requests Application** is a key component designed to handle GitHub requests in a secure and consistent manner by incorporating automation with Terraform (repository [here](https://github.com/cabinetoffice/github-requests-terraform)) and implementing security access through the COLA Identity Provider.
For example, when a new member wishes to add their GitHub account to a team, they submit an issue through this app. The IDP team verifies the issues, applies changes, and merges the Pull Request to the main branch. On the next Terraform run, the changes propagate to GitHub, granting access. This entire process occurs with complete visibility within the department, ensuring consistency, instead of relying on a single user to make changes through GitHub's web interface.

## Workflow

1. User makes a request through the GitHub application.
2. After authentication with COLA, user's request starts creating the data object.
3. The user fills in all the details and submits the object from the `Check Your Answers` page, triggering:
1. Terraform Repository: An issue is created.
2. DynamoDB: User information is saved.
3. Email Service: An email is sent to the user to confirm the information sent.
4. Platform team assesses the issue in the Terraform repository.
1. Upon readiness, a plan and apply are triggered.
2. DynamoDB database is accessed for user information if needed (e.g., to send email to user with expired contract).
5. Notify sends confirmation or any updates to the user.

## Overview and Design

The GitHub Requests Application is a Node.js-based web application that provides a simple interface. Internal users (members of the Cognito user pool) fill in forms with requested details. When a request is submitted, an issue is created on the dedicated terraform repository, and an email is sent to the user from our `github-request.idp` email address. The email includes a message containing the user's filled-in information.

The issue is then reviewed by the team, and further comments may be requested if necessary. Approval must be granted by two members of the team to avoid misconfigurations.

### Sending an Email

The Node.js app will send the email with Amazon SES, and you can find the implementation details [here](https://docs.aws.amazon.com/sdk-for-javascript/v2/developer-guide/ses-examples-sending-email.html#ses-examples-sendmail), in particular we will create an object to pass the parameter values that define the email to be sent, including sender and receiver addresses, subject, and email body in plain text (with optional HTML).

### Github Issue

The issue will be created by using the CO `@co-digital/api-sdk` module. It has already been tested locally with the following snippets (eg [here](https://github.com/Mouhajer-CO/git-api-calls/blob/14f7bd33e9ac579fead0b2003c35946a5b0cacf7/src/utils.ts#L43)):

```js
export const createIssue = async (title: string, description: string) => {
try {
const url = `<https://api.github.com/repos/${OWNER}/${REPOS}/issues`;>
const body = { title, body: description, assignees: [ASSIGNEE], labels: [LABEL] };

return await fetchCall(url, body);
} catch (error: any) {
console.error('Error:', error);
}
}
```

### Terraform Code

We will utilize Terraform and the GitHub provider to create resources and invite users to the GitHub organization by defining separate JSON files. Possible implementation details can be found [here](https://developer.hashicorp.com/terraform/tutorials/it-saas/github-user-teams). We will use JSON file data with the following function to retrieve information: `jsondecode(file("${path.module}/teams.json"))`. We could use an S3 bucket and push members/teams/repos data before terraform apply.

Initially, we need to use the `terraform import` command to import existing infrastructure into terraform, utilizing JSON files for members, teams, repositories, members per team, and so on. This configuration scales better and enables the team to follow best practices safely, consistently, and efficiently.

### Set up

- Import repos, teams and members details through the api sdk node project
- Set `.tf` configuration file for your organization as json files on datasets folder
- Run basic commands `terraform init`, `terraform fmt` and `terraform validate`
- Set Resources configurations in the root module, it will be removed later after state file generated when resources imported.

```t
resource "github_repository" "terraform-modules" {
# (resource arguments)
}
```

- Import the actual repository using terraform import github_repository.your-repo your-repo
- Run terraform plan to ensure that your configuration matches what is on GitHub, or change the configuration until it matches.
- Run terraform apply.
- Remove ownership to members not part of terraform platform team (at least five people)

## Benefits

- **Programmatic Approach**: Implementation of a systematic and automated process.
- **Authorized Users**: All users are part of defined Cognito user pools, ensuring authorized access.
- **Detailed Information**: Comprehensive information from required UI inputs (eg. distinguishing between long and short-term users).
- **Least Privilege Access to GitHub UI**: Minimized access in the GitHub UI to maintain security (limiting OWNER roles to at least two members, no more then five).
- **Decreased Risk of Errors**: Reduction in errors compared to manual UI interactions, minimizing the potential for mistakes.
- **Increased Visibility**: All changes are reviewed and tracked in version control, involving at least two team members.
- **Increased Consistency**: Uniform application of changes across all repositories, addressing issues such as the absence of branch protection and team maintainers.
- **Single Source of Truth**: Changes are exclusively applied through Terraform, eliminating the need for manual UI alterations.

## Possible issues

- The GitHub API appears to have slow performance, and the terraform apply process can take 30 minutes or more for large organizations, as mentioned in this [issue](https://github.com/integrations/terraform-provider-github/issues/567). To address this, we could consider using a Node.js script to fetch repository data ahead of time to optimize terraform timing.

- How can we ensure that the user pool contains only the required users? All internal members are allowed, and there must be at least one maintainer per team, as well as per repository. External team, contractor, or any other outsourcing company have to be part of the Cognito Users pool, and they need to request access ahead of time. Update to new entry wiki page has to be done.

- The Terraform configuration serves as the single source of truth and policy for most of Github requests. No manual changes should be made to prevent configuration inconsistencies. Owners have to (and will) be reduced.

- To reduce seats, maintainers for all users need to be identified.

- Importing existing infrastructure might be time-consuming, especially if we need to remove unused teams, members, or repositories, and the terraform apply process must be carefully defined the first time to avoid issues (backup in place?).
Loading

0 comments on commit 05d6795

Please sign in to comment.