feat(server, builder): OAuth support + API key management + GitHub blocks #8044

Pwuts · 2024-09-12T00:07:29Z

Implement front-to-back GitHub OAuth support on blocks #8002

Changes 🏗️

Config

For Supabase, the back end needs SUPABASE_URL, SUPABASE_SERVICE_KEY, and SUPABASE_JWT_SECRET
- SUPABASE_SERVICE_KEY must be a service role key as the anon key doesn't have sufficient privileges for the operations it's used for
For the GitHub integration to work, the back end needs GITHUB_CLIENT_ID and GITHUB_CLIENT_SECRET

Front end

UI components

CredentialsInput for use on CustomNode: allows user to add/select credentials for a service.
- APIKeyCredentialsModal: a dialog for creating API keys
- OAuth2FlowWaitingModal: a dialog to indicate that the application is waiting for the user to log in to the 3rd party service in the provided pop-up window
NodeCredentialsInput: wrapper for CredentialsInput with the "usual" interface of node input components
New icons: IconKey, IconKeyPlus, IconUser, IconUserPlus

Data model

CredentialsProvider: introduces the app-level CredentialsProvidersContext, which acts as an application-wide store and cache for credentials metadata.
useCredentials for use on CustomNode: uses CredentialsProvidersContext and provides node-specific credential data and provider-specific data/functions
/auth/integrations/oauth_callback route to close the loop to the CredentialsInput after a user completes sign-in to the external service
Add BlockIOCredentialsSubSchema

API client

Add isAuthenticated method
Add methods for integration OAuth flow: oAuthLogin, oAuthCallback
Add CRD methods for credentials: createAPIKeyCredentials, listCredentials, getCredentials, deleteCredentials
Add mirrored types CredentialsMetaResponse, CredentialsMetaInput, OAuth2Credentials, APIKeyCredentials

REST API

Tweak output of OAuth /login endpoint: add state_token separately in response
Add POST /integrations/{provider}/credentials (for API keys)
Add DELETE /integrations/{provider}/credentials/{cred_id}

Back end

Add Supabase support to AppService
Add FRONTEND_BASE_URL config option, mainly for local development use

`autogpt_libs.supabase_integration_credentials_store`

Add CredentialsType alias
Add .bearer() helper methods to APIKeyCredentials and OAuth2Credentials

Blocks

Add CredentialsField(..) -> CredentialsMetaInput
Add GitHub blocks + "DEVELOPER_TOOLS" category
Add **kwargs to Block.run(..) signature to support additional kwargs

Executor

Add strict support for credentials fields on blocks
Fetch credentials for graph execution and pass them down through to the node execution

PR Quality Scorecard ✨

Have you used the PR description template? +2 pts
Is your pull request atomic, focusing on a single change? +5 pts
Have you linked the GitHub issue(s) that this PR addresses? +5 pts
Have you documented your changes clearly and comprehensively? +5 pts
Have you changed or added a feature? -4 pts
- Have you added/updated corresponding documentation? +4 pts
- Have you added/updated corresponding integration tests? +5 pts
Have you changed the behavior of AutoGPT? -5 pts
- Have you also run agbenchmark to verify that these changes do not regress performance? +10 pts

…github-oauth-support-on-blocks

…pe` alias

…own through to the node execution

…rastructure

github-actions · 2024-09-12T00:07:42Z

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

netlify · 2024-09-12T00:08:28Z

✅ Deploy Preview for auto-gpt-docs canceled.

Name	Link
🔨 Latest commit	`6aea79b`
🔍 Latest deploy log	https://app.netlify.com/sites/auto-gpt-docs/deploys/66ec5dbef6a1020008f2e971

…github-oauth-support-on-blocks

github-actions · 2024-09-12T10:51:02Z

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

…github-oauth-support-on-blocks

github-actions · 2024-09-12T11:11:38Z

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

rnd/autogpt_server/autogpt_server/blocks/basic.py

kcze · 2024-09-12T13:06:56Z

rnd/autogpt_server/autogpt_server/blocks/github.py

ugh, how about making a blocks/github directory and splitting this up a bit?

Can do later. For now it's easier to mass-edit them if they're all in one file.

rnd/autogpt_server/autogpt_server/data/block.py

github-actions · 2024-09-14T16:49:46Z

This pull request has conflicts with the base branch, please resolve those so we can evaluate the pull request.

…github-oauth-support-on-blocks

github-actions · 2024-09-16T13:51:59Z

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

…github-oauth-support-on-blocks

github-actions · 2024-09-18T21:55:47Z

Conflicts have been resolved! 🎉 A maintainer will review the pull request shortly.

…github-oauth-support-on-blocks

## Front end ### UI components - `CredentialsInput` for use on `CustomNode`: allows user to add/select credentials for a service. - `APIKeyCredentialsModal`: a dialog for creating API keys - `OAuth2FlowWaitingModal`: a dialog to indicate that the application is waiting for the user to log in to the 3rd party service in the provided pop-up window - `NodeCredentialsInput`: wrapper for `CredentialsInput` with the "usual" interface of node input components ### Data model - `CredentialsProvider`: introduces the app-level `CredentialsProvidersContext`, which acts as an application-wide store and cache for credentials metadata. - `useCredentials` for use on `CustomNode`: uses `CredentialsProvidersContext` and provides node-specific credential data and provider-specific data/functions - `/auth/integrations/oauth_callback` route to close the loop to the `CredentialsInput` after a user completes sign-in to the external service ### API client - Add `oAuthLogin`, `oAuthCallback` methods - Add types `BlockIOCredentialsSubSchema`, `CredentialsMetaInput` - Add `isAuthenticated` method ## REST API - Tweak output of OAuth `/login` endpoint: add `state_token` separately in response ## Back end - Add `required_scopes` param to `CredentialsField` (`credentials_scopes` in schema) - Add scope requirement (`repo` in all cases) to all GitHub blocks - Add `FRONTEND_BASE_URL` config option, mainly for local development use. --------- Co-authored-by: Reinier van der Leer <[email protected]>

Pwuts · 2024-09-19T02:26:05Z

If you read this, this PR works :)

aarushik93 · 2024-09-19T13:15:12Z

@Pwuts how do I run this PR? Are there config settings for supabase needed?

Pwuts · 2024-09-19T13:21:06Z

@aarushik93 I have updated the description with additional config instructions:

For Supabase, the back end now needs SUPABASE_URL, SUPABASE_KEY, and SUPABASE_JWT_SECRET

SUPABASE_KEY must be a service role key as the anon key doesn't have sufficient privileges for the operations it's used for

For the GitHub integration to work, the back end now needs GITHUB_CLIENT_ID and GITHUB_CLIENT_SECRET

aarushik93 · 2024-09-19T13:30:04Z

Thanks could you please add the new envrionment variables (SUPABASE_KEY) to the appropriate places in the .env.example and docker compose please?

aarushik93 · 2024-09-19T13:32:06Z

docs/content/server/new_blocks.md

@@ -84,7 +84,7 @@ Follow these steps to create and test a new block:
 5. **Implement the `run` method with error handling:**, this should contain the main logic of the block:



For the GitHub integration to work, the back end now needs GITHUB_CLIENT_ID and GITHUB_CLIENT_SECRET

In the docs and readme, can we add a section for "integration blocks" or wtv we are calling them and start adding this kind of information about configuring them?

Done: 74138d7

I didn't update the README because:

I want to avoid duplicating documentation

The extra stuff for credentials technically fits within the summary instructions for adding blocks that's currently in the README

thanks - what about also adding somewhere - where I go to find client id and secret and API key?
I was thinking we'd just have a subsection somewhere in docs for each integration that just tells you where to go

cause rn I can't find exactly where to go for the gh client id and secret....

Added to .env.example: 57afc85

aarushik93 · 2024-09-19T13:34:14Z

rnd/autogpt_server/autogpt_server/data/block.py

@@ -36,6 +38,7 @@ class BlockCategory(Enum):
    INPUT = "Block that interacts with input of the graph."
    OUTPUT = "Block that interacts with output of the graph."
    LOGIC = "Programming logic to control the flow of your agent"
+    DEVELOPER_TOOLS = "Developer tools like Github."  # added this so all the github blocks are in the same category for now, this is to be changed


what needs to be changed? The desc or the name? I quite like the category name tbh

It was a suggestion to maybe change the name or description if needed but if its fine and does not need changing then the comment can be removed and it can be kept as is 😄

Bentlybro and others added 10 commits September 10, 2024 19:17

Add Github blocks + "DEVELOPER_TOOLS" category

efdd71b

Merge branch 'master' into reinier/open-1806-implement-front-to-back-…

57c22e0

…github-oauth-support-on-blocks

move github_blocks.py -> github.py

4cec829

remove duplicate OAuth handler files

e7fa682

feat(server): Add Supabase support to AppService

8ae5378

Add **kwargs to Block.run(..) signature to support additional kwargs

20c5137

feat(libs/supabase_integration_credentials_store): Add `CredentialsTy…

08666f4

…pe` alias

Add strict support for credentials fields on blocks

14c8ca4

feat(executor): Fetch credentials for graph execution and pass them d…

da6d443

…own through to the node execution

feat(server/blocks): Integrate GitHub blocks with new credentials inf…

56d379f

…rastructure

github-actions bot added conflicts Automatically applied to PRs with merge conflicts documentation Improvements or additions to documentation Server size/xl and removed conflicts Automatically applied to PRs with merge conflicts labels Sep 12, 2024

Merge branch 'master' into reinier/open-1806-implement-front-to-back-…

d7e34b1

…github-oauth-support-on-blocks

github-actions bot added the conflicts Automatically applied to PRs with merge conflicts label Sep 12, 2024

Merge branch 'master' into reinier/open-1806-implement-front-to-back-…

c623d61

…github-oauth-support-on-blocks

github-actions bot removed the conflicts Automatically applied to PRs with merge conflicts label Sep 12, 2024

kcze reviewed Sep 12, 2024

View reviewed changes

github-actions bot added the conflicts Automatically applied to PRs with merge conflicts label Sep 14, 2024

Merge branch 'master' into reinier/open-1806-implement-front-to-back-…

26a837a

…github-oauth-support-on-blocks

github-actions bot removed the conflicts Automatically applied to PRs with merge conflicts label Sep 16, 2024

Fix schema output with extra properties on complex input fields

c8f0110

github-actions bot added the Builder AutoGPT Builder label Sep 18, 2024

Merge branch 'master' into reinier/open-1806-implement-front-to-back-…

03fe5a1

…github-oauth-support-on-blocks

github-actions bot removed the conflicts Automatically applied to PRs with merge conflicts label Sep 18, 2024

Pwuts and others added 3 commits September 19, 2024 00:05

fix conflict resolve in integrations API router

f4a92dc

Merge branch 'master' into reinier/open-1806-implement-front-to-back-…

c8115b1

…github-oauth-support-on-blocks

Pwuts marked this pull request as ready for review September 19, 2024 02:16

Pwuts requested a review from a team as a code owner September 19, 2024 02:16

Pwuts requested review from Bentlybro and aarushik93 and removed request for a team September 19, 2024 02:16

fix field description on API key dialog

d9b2ccc

Pwuts changed the title ~~feat(server, builder): Add GitHub blocks with OAuth support~~ feat(server, builder): OAuth support + API key management + GitHub blocks Sep 19, 2024

parametrize test_available_blocks

e260c39

aarushik93 reviewed Sep 19, 2024

View reviewed changes

Update .env.example and docker-compose.yml with new config variables

a5264d6

Pwuts force-pushed the reinier/open-1806-implement-front-to-back-github-oauth-support-on-blocks branch 2 times, most recently from 04143db to 47786f6 Compare September 19, 2024 15:28

add docs for adding authenticated blocks

74138d7

Pwuts force-pushed the reinier/open-1806-implement-front-to-back-github-oauth-support-on-blocks branch from 47786f6 to 74138d7 Compare September 19, 2024 15:34

Pwuts requested a review from aarushik93 September 19, 2024 15:35

Pwuts added 2 commits September 19, 2024 18:27

add comment to .env.example about GitHub app type

57afc85

fix block tests with credentials

6aea79b

Pwuts force-pushed the reinier/open-1806-implement-front-to-back-github-oauth-support-on-blocks branch from 5c83d97 to 6aea79b Compare September 19, 2024 17:22

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(server, builder): OAuth support + API key management + GitHub blocks #8044

feat(server, builder): OAuth support + API key management + GitHub blocks #8044

Pwuts commented Sep 12, 2024 •

edited

Loading

github-actions bot commented Sep 12, 2024

netlify bot commented Sep 12, 2024 •

edited

Loading

github-actions bot commented Sep 12, 2024

github-actions bot commented Sep 12, 2024

kcze Sep 12, 2024

Pwuts Sep 12, 2024

github-actions bot commented Sep 14, 2024

github-actions bot commented Sep 16, 2024

github-actions bot commented Sep 18, 2024

Pwuts commented Sep 19, 2024

aarushik93 commented Sep 19, 2024

Pwuts commented Sep 19, 2024 •

edited

Loading

aarushik93 commented Sep 19, 2024

aarushik93 Sep 19, 2024 •

edited

Loading

Pwuts Sep 19, 2024 •

edited

Loading

Pwuts Sep 19, 2024

aarushik93 Sep 19, 2024

aarushik93 Sep 19, 2024

Pwuts Sep 19, 2024

aarushik93 Sep 19, 2024

Pwuts Sep 19, 2024

Bentlybro Sep 19, 2024

		@@ -84,7 +84,7 @@ Follow these steps to create and test a new block:
		5. Implement the `run` method with error handling:, this should contain the main logic of the block:

feat(server, builder): OAuth support + API key management + GitHub blocks #8044

Are you sure you want to change the base?

feat(server, builder): OAuth support + API key management + GitHub blocks #8044

Conversation

Pwuts commented Sep 12, 2024 • edited Loading

Changes 🏗️

Config

Front end

UI components

Data model

API client

REST API

Back end

autogpt_libs.supabase_integration_credentials_store

Blocks

Executor

PR Quality Scorecard ✨

github-actions bot commented Sep 12, 2024

netlify bot commented Sep 12, 2024 • edited Loading

✅ Deploy Preview for auto-gpt-docs canceled.

github-actions bot commented Sep 12, 2024

github-actions bot commented Sep 12, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Sep 14, 2024

github-actions bot commented Sep 16, 2024

github-actions bot commented Sep 18, 2024

Pwuts commented Sep 19, 2024

aarushik93 commented Sep 19, 2024

Pwuts commented Sep 19, 2024 • edited Loading

aarushik93 commented Sep 19, 2024

aarushik93 Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

Pwuts Sep 19, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Pwuts commented Sep 12, 2024 •

edited

Loading

`autogpt_libs.supabase_integration_credentials_store`

netlify bot commented Sep 12, 2024 •

edited

Loading

Pwuts commented Sep 19, 2024 •

edited

Loading

aarushik93 Sep 19, 2024 •

edited

Loading

Pwuts Sep 19, 2024 •

edited

Loading