Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

added local minio s3 and backup:mongodb script #649

Merged
merged 6 commits into from
Mar 6, 2024
Merged

added local minio s3 and backup:mongodb script #649

merged 6 commits into from
Mar 6, 2024

Conversation

ARtheboss
Copy link

No description provided.

kevinzwang
kevinzwang requested changes Nov 7, 2023
View reviewed changes
Copy link
Member

@kevinzwang kevinzwang left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good. I'd also like to see a restore script (with parameters for the collections to restore) so we can test this end-to-end before merge.

Also, would it be possible to include lifecycle configurations so that the backups automatically delete after a certain time frame? See https://docs.aws.amazon.com/AmazonS3/latest/API/API_PutBucketLifecycleConfiguration.html

backend/src/config.ts Outdated
Comment on lines 47 to 51
S3_ENDPOINT: env("S3_ENDPOINT"),
S3_PORT: parseInt(env("S3_PORT")),
S3_ACCESS_KEY_ID: env("S3_ACCESS_KEY_ID"),
S3_SECRET_ACCESS_KEY: env("S3_SECRET_ACCESS_KEY"),
S3_MONGO_BACKUP_BUCKET: env("S3_MONGO_BACKUP_BUCKET"),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

maybe put these under an s3 key like config.mongoDB

backend/src/scripts/backup-mongodb.ts Show resolved Hide resolved
@ARtheboss
Copy link
Author

@kevinzwang added your requested changes

@ARtheboss ARtheboss linked an issue Feb 7, 2024 that may be closed by this pull request
Write a script to backup to Google Cloud #598
Open
@ARtheboss
Copy link
Author

@kevinzwang should be good to go

@kevinzwang kevinzwang self-requested a review February 23, 2024 03:00
kevinzwang
kevinzwang approved these changes Feb 23, 2024
View reviewed changes
Copy link
Member

@kevinzwang kevinzwang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. Resolve your conflicts and then pls squash and merge!

@ARtheboss ARtheboss merged commit d6959a2 into gql Mar 6, 2024
1 check passed
@ARtheboss ARtheboss deleted the mongodb-backup branch March 6, 2024 22:37
@maxmwang maxmwang restored the mongodb-backup branch March 17, 2024 21:23
maxmwang added a commit that referenced this pull request Mar 17, 2024
@maxmwang
Revert "added local minio s3 and backup:mongodb script (#649)"
233a83c 
This reverts commit d6959a2.
@maxmwang maxmwang mentioned this pull request Mar 17, 2024
Merged
maxmwang pushed a commit that referenced this pull request Jul 22, 2024
@ARtheboss @maxmwang
added local minio s3 and backup:mongodb script (#649)
515a038 
* added local minio s3 and backup:mongodb script

* included metadata.json files to backup

* added download code

* added lifecycle and changed config names

* removed minio container, added lifecycle rule check

---------

Co-authored-by: ARtheboss <[email protected]>
maxmwang added a commit that referenced this pull request Jul 22, 2024
@maxmwang
Revert "added local minio s3 and backup:mongodb script (#649)"
8b7bd0a 
This reverts commit bc514c0.
maxmwang added a commit that referenced this pull request Jul 22, 2024
@maxmwang
Squashed commit of the following:
4c1854d 
commit 48acfe9
Author: Matthew Rowland <[email protected]>
Date:   Tue Jun 25 10:56:11 2024 -0700

    fix: Allow frontend to build

commit 6b54422
Merge: 7e730ef 419ed5b
Author: Matthew Rowland <[email protected]>
Date:   Tue Jun 25 09:43:07 2024 -0700

    Merge pull request #711 from asuc-octo/frontend-rewrite

    Merge frontend work into GQL branch

commit 419ed5b
Merge: 735aaf4 7e730ef
Author: Matthew Rowland <[email protected]>
Date:   Mon Jun 24 16:44:56 2024 -0700

    Merge branch 'gql' into frontend-rewrite

commit 735aaf4
Author: Matthew Rowland <[email protected]>
Date:   Mon Jun 24 11:56:26 2024 -0700

    feat: Winter, GZIP compression, getCourseList

commit 7e730ef
Author: Max Wang <[email protected]>
Date:   Sun Jun 23 22:50:10 2024 -0700

    feat: use redis for user session cookies (#710)

commit 271469a
Author: Matthew Rowland <[email protected]>
Date:   Tue Jun 18 12:58:05 2024 -0700

    feat: Optional query persistence for Browser, internal Browser API fetching

commit 5e5c905
Author: Matthew Rowland <[email protected]>
Date:   Sun Jun 16 13:02:12 2024 -0700

    feat: Init schedules

commit 56c7676
Author: Matthew Rowland <[email protected]>
Date:   Sun Jun 16 01:14:54 2024 -0700

    feat: Log in and log out with redirectURI via base64-encoded state

commit 5ed2234
Merge: 81e9329 3b5ee90
Author: Matthew Rowland <[email protected]>
Date:   Sun Jun 16 00:22:55 2024 -0700

    Merge branch 'gql' into frontend-rewrite

commit 3b5ee90
Author: maxmwang <[email protected]>
Date:   Sun Jun 16 00:05:15 2024 -0700

    fix user auth

commit 7a40c8b
Author: maxmwang <[email protected]>
Date:   Sat Jun 15 23:58:55 2024 -0700

    revert

commit ef5a8ad
Author: maxmwang <[email protected]>
Date:   Sat Jun 15 23:40:55 2024 -0700

    fix: add placeholder value for refresh_token

commit 81e9329
Author: Matthew Rowland <[email protected]>
Date:   Tue Jun 11 18:13:54 2024 -0700

    feat: Filter by day

commit b35bdcb
Author: Matthew Rowland <[email protected]>
Date:   Sun Jun 9 19:53:24 2024 -0700

    feat: Tooltip

commit bc02bf4
Author: Matthew Rowland <[email protected]>
Date:   Sun Jun 9 14:49:10 2024 -0700

    feat: Capacity color

commit 756b343
Author: Matthew Rowland <[email protected]>
Date:   Sun Jun 9 14:45:05 2024 -0700

    feat: Schedule context (units, classes, term)

commit 6af982a
Author: Matthew Rowland <[email protected]>
Date:   Sun Jun 9 11:50:25 2024 -0700

    feat: Manage

commit eee78be
Author: Matthew Rowland <[email protected]>
Date:   Fri Jun 7 17:51:27 2024 -0700

    feat: Compare

commit 41ed7e6
Author: Matthew Rowland <[email protected]>
Date:   Fri Jun 7 12:26:14 2024 -0700

    feat: Schedule, Calendar

commit f701d01
Author: Matthew Rowland <[email protected]>
Date:   Sun Jun 2 13:58:40 2024 -0700

    feat: Filtering

commit 5f34a05
Author: Matthew Rowland <[email protected]>
Date:   Tue May 7 10:03:58 2024 -0700

    feat: Enrollment, Grades

commit 8be7703
Author: Matthew Rowland <[email protected]>
Date:   Sun May 5 22:12:08 2024 -0700

    feat: In-progress migration to expanded data formats

commit 1a54ba5
Author: Matthew Rowland <[email protected]>
Date:   Sun Apr 28 21:59:11 2024 -0700

    feat: Reserved seats

commit 4297d42
Author: Matthew Rowland <[email protected]>
Date:   Sat Apr 27 22:09:18 2024 -0700

    feat: Sorting

commit ed8020d
Author: Matthew Rowland <[email protected]>
Date:   Thu Apr 25 00:09:37 2024 -0700

    feat: Filters

commit d1968a9
Author: Matthew Rowland <[email protected]>
Date:   Tue Apr 23 21:24:43 2024 -0700

    fix: uri

commit 7bc1fff
Author: Matthew Rowland <[email protected]>
Date:   Tue Apr 23 21:19:12 2024 -0700

    chore: gitignore

commit e55ea83
Author: Matthew Rowland <[email protected]>
Date:   Tue Apr 23 21:16:36 2024 -0700

    fix: dist

commit 297755c
Author: Matthew Rowland <[email protected]>
Date:   Tue Apr 23 21:14:01 2024 -0700

    fix: start

commit dcd6a49
Author: Matthew Rowland <[email protected]>
Date:   Tue Apr 23 21:04:24 2024 -0700

    fix: types

commit 647c706
Author: Matthew Rowland <[email protected]>
Date:   Tue Apr 23 20:58:27 2024 -0700

    feat: anyPrintInScheduleOfClasses

commit b221fc0
Author: Matthew Rowland <[email protected]>
Date:   Tue Apr 23 20:21:42 2024 -0700

    feat: usePrefersColorScheme

commit 110c962
Author: Matthew Rowland <[email protected]>
Date:   Sun Apr 21 21:41:27 2024 -0700

    feat: Track markers

commit 4b1da35
Author: Matthew Rowland <[email protected]>
Date:   Sun Apr 21 21:35:25 2024 -0700

    feat: Preview sections, Itinerary

commit 12530e2
Author: Matthew Rowland <[email protected]>
Date:   Sun Apr 21 14:39:41 2024 -0700

    feat: section

commit 7a81c36
Author: Matthew Rowland <[email protected]>
Date:   Sat Apr 20 12:01:38 2024 -0700

    feat: Search

commit 2e660cf
Author: ARtheboss <[email protected]>
Date:   Tue Apr 16 21:50:26 2024 -0700

    fixes for course/catalog: added sections list, list type corrections, course grades

commit ce16be2
Author: Matthew Rowland <[email protected]>
Date:   Fri Apr 19 16:23:50 2024 -0700

    feat: Search

commit 3a41c7e
Merge: 6f891fe 1145edd
Author: Matthew Rowland <[email protected]>
Date:   Wed Apr 17 15:32:38 2024 -0700

    Merge branch 'gql' into frontend-rewrite

commit 6f891fe
Author: Matthew Rowland <[email protected]>
Date:   Wed Apr 17 15:31:48 2024 -0700

    feat: Init plan

commit 1145edd
Merge: 78b8cb2 a7a95a4
Author: Max Wang <[email protected]>
Date:   Tue Apr 16 21:22:36 2024 -0700

    Merge pull request #705 from asuc-octo/gql-apollo-caching

    New Backend Caching

commit a7a95a4
Author: Kevin Wang <[email protected]>
Date:   Tue Apr 16 21:01:41 2024 -0700

    remove extranneous cacheControl directive

commit bcd9ede
Merge: a9c289a 78b8cb2
Author: Matthew Rowland <[email protected]>
Date:   Tue Apr 16 10:01:55 2024 -0700

    Merge branch 'gql' into frontend-rewrite

commit a9c289a
Author: Matthew Rowland <[email protected]>
Date:   Tue Apr 16 10:01:09 2024 -0700

    feat: Support old terms, move filtering to Browser, Dockerfile, etc.

commit 8727662
Author: Matthew Rowland <[email protected]>
Date:   Mon Apr 15 00:49:47 2024 -0700

    feat: Units, responsive Details, etc.

commit cc05d01
Author: Kevin Wang <[email protected]>
Date:   Sun Apr 14 23:57:19 2024 -0700

    add caching to apollo queries

commit 5f51ac2
Author: Matthew Rowland <[email protected]>
Date:   Sun Apr 14 21:13:07 2024 -0700

    feat: Plan, responsive Browser, responsive Catalog, responsive Calendar

commit 9f5cb2d
Author: Matthew Rowland <[email protected]>
Date:   Sat Apr 13 15:23:38 2024 -0700

    feat: Dark theme, search for and add classes to a schedule, etc.

commit a933adf
Author: Matthew Rowland <[email protected]>
Date:   Fri Apr 12 17:02:01 2024 -0700

    feat: Location, Time, Details, CCN, Tooltip; sections, etc.

commit 78b8cb2
Merge: 492759b 363a02b
Author: Max Wang <[email protected]>
Date:   Thu Apr 11 14:24:58 2024 -0700

    Merge pull request #691 from asuc-octo/mathhulk-patch-1

    Update controller.ts

commit 363a02b
Author: maxmwang <[email protected]>
Date:   Thu Apr 11 14:23:48 2024 -0700

    TODO(production) flag

commit 492759b
Author: maxmwang <[email protected]>
Date:   Thu Apr 11 14:22:38 2024 -0700

    TODO(production) flag

commit b1ceba7
Author: maxmwang <[email protected]>
Date:   Thu Apr 11 12:40:11 2024 -0700

    allow introspection on stanfurdtime

commit 2b11afa
Author: maxmwang <[email protected]>
Date:   Tue Apr 9 21:10:19 2024 -0700

    fix github actions

commit 643ff8b
Author: maxmwang <[email protected]>
Date:   Tue Apr 9 21:07:50 2024 -0700

    update github actions from old repo configs

commit f334238
Merge: 5ed4ccc 4269b7e
Author: Max Wang <[email protected]>
Date:   Tue Apr 9 20:35:51 2024 -0700

    Merge pull request #704 from asuc-octo/gql-updated/updatedat-fix

    added resiliency against variance between updated and updatedAt

commit 4269b7e
Author: ARtheboss <[email protected]>
Date:   Thu Apr 4 19:51:19 2024 -0700

    added resiliency against variance between updated and updatedAt

commit e0e667a
Author: Matthew Rowland <[email protected]>
Date:   Thu Mar 21 12:49:51 2024 -0700

    feat: Init map markers, look into directions

commit fcb7760
Author: Matthew Rowland <[email protected]>
Date:   Tue Mar 19 16:06:51 2024 -0700

    feat: campus.geojson

commit 2150698
Author: Matthew Rowland <[email protected]>
Date:   Tue Mar 19 00:04:56 2024 -0700

    feat: Buildings

commit c27889e
Author: Matthew Rowland <[email protected]>
Date:   Sun Mar 17 23:15:32 2024 -0700

    feat: Itinerary

commit ee9084c
Merge: 1ba33c2 5ed4ccc
Author: maxmwang <[email protected]>
Date:   Sun Mar 17 14:24:13 2024 -0700

    Merge branch 'gql' into mathhulk-patch-1

commit 5ed4ccc
Merge: bc514c0 1f3829c
Author: Max Wang <[email protected]>
Date:   Sun Mar 17 14:23:26 2024 -0700

    Merge pull request #692 from asuc-octo/revert-649-mongodb-backup

    Revert "added local minio s3 and backup:mongodb script"

commit 1f3829c
Author: Max Wang <[email protected]>
Date:   Sun Mar 17 14:23:13 2024 -0700

    Revert "added local minio s3 and backup:mongodb script (#649)"

    This reverts commit bc514c0.

commit 1ba33c2
Author: Matthew Rowland <[email protected]>
Date:   Sun Mar 17 01:07:24 2024 -0700

    Update controller.ts

commit 496e023
Author: Matthew Rowland <[email protected]>
Date:   Thu Mar 14 23:46:54 2024 -0700

    feat: Init dynamic calendar

commit 45276e1
Author: Matthew Rowland <[email protected]>
Date:   Thu Mar 14 19:58:54 2024 -0700

    feat: Init events

commit 11ca8c1
Author: Matthew Rowland <[email protected]>
Date:   Thu Mar 14 17:22:55 2024 -0700

    feat: Init Schedule

commit bc514c0
Author: ARtheboss <[email protected]>
Date:   Wed Mar 6 14:37:19 2024 -0800

    added local minio s3 and backup:mongodb script (#649)

    * added local minio s3 and backup:mongodb script

    * included metadata.json files to backup

    * added download code

    * added lifecycle and changed config names

    * removed minio container, added lifecycle rule check

    ---------

    Co-authored-by: ARtheboss <[email protected]>

commit c099703
Author: Matthew Rowland <[email protected]>
Date:   Sun Mar 3 18:15:52 2024 -0800

    feat: Tooltips, Saturday + Sunday

commit 59f8e00
Author: Matthew Rowland <[email protected]>
Date:   Sun Mar 3 15:54:44 2024 -0800

    feat: Empty placeholder, mock Calendar

commit 02a26a3
Author: Matthew Rowland <[email protected]>
Date:   Sun Mar 3 13:23:34 2024 -0800

    feat: Filtering

commit 1040a16
Author: Matthew Rowland <[email protected]>
Date:   Sat Mar 2 18:42:45 2024 -0800

    feat: Class

commit f657db8
Merge: eaf4312 76eb98c
Author: maxmwang <[email protected]>
Date:   Thu Feb 29 01:19:34 2024 -0800

    Merge branch 'gql-proxy-auth' into gql

commit eaf4312
Author: maxmwang <[email protected]>
Date:   Thu Feb 29 01:12:29 2024 -0800

    clean up root

commit 76eb98c
Author: maxmwang <[email protected]>
Date:   Thu Feb 29 01:08:00 2024 -0800

    express trust proxy (nginx)

commit 5577f2b
Author: maxmwang <[email protected]>
Date:   Thu Feb 29 00:54:28 2024 -0800

    fix catalog query with course model change

commit 3889271
Merge: 16fb4f9 87dc2a9
Author: Max Wang <[email protected]>
Date:   Thu Feb 29 00:25:09 2024 -0800

    Merge pull request #682 from asuc-octo/gql-cors

    Allow local frontend to authenticate

commit e07b62c
Author: Matthew Rowland <[email protected]>
Date:   Thu Feb 29 00:20:23 2024 -0800

    feat: Initialize AccountContext, Features images, etc.

commit 87dc2a9
Author: Matthew Rowland <[email protected]>
Date:   Wed Feb 28 19:39:16 2024 -0800

    feat: localhost:3001

commit 16fb4f9
Author: maxmwang <[email protected]>
Date:   Mon Feb 26 19:25:12 2024 -0800

    add collections name to mongo models

commit 9c31b41
Author: Matthew Rowland <[email protected]>
Date:   Sun Feb 4 19:41:46 2024 -0800

    feat: Course

commit 6464bc5
Author: Matthew Rowland <[email protected]>
Date:   Sat Feb 3 17:26:07 2024 -0800

    feat: Landing

commit dabc4da
Author: Matthew Rowland <[email protected]>
Date:   Sat Feb 3 14:36:59 2024 -0800

    feat: Initialize

commit fc1af7a
Author: maxmwang <[email protected]>
Date:   Mon Nov 13 19:16:48 2023 -0800

    revert: use POST for logout

commit c95328c
Author: Chengming-Li <[email protected]>
Date:   Mon Nov 13 18:36:30 2023 -0800

    Rewrite schedule schema to better track courses(#644)

    * fix setSelectedSections bug

    * change mongo schema and typedefs

    * more progress

    * even more progress

    * finish redesign

    * housekeeping

    * fix bugs

commit 262bba2
Author: maxmwang <[email protected]>
Date:   Mon Nov 13 18:26:58 2023 -0800

    fix: remove Makefile

commit 2d6259e
Merge: 4d22a5f 12738eb
Author: Max Wang <[email protected]>
Date:   Thu Nov 9 04:11:14 2023 -0800

    Merge pull request #651 from asuc-octo/gql-logout

    Logout route fix

commit 12738eb
Author: maxmwang <[email protected]>
Date:   Mon Nov 6 20:23:19 2023 -0800

    fix: change logout route to GET
maxmwang pushed a commit that referenced this pull request Jul 22, 2024
@ARtheboss
added local minio s3 and backup:mongodb script (#649)
bc514c0 
* added local minio s3 and backup:mongodb script

* included metadata.json files to backup

* added download code

* added lifecycle and changed config names

* removed minio container, added lifecycle rule check

---------

Co-authored-by: ARtheboss <[email protected]>
maxmwang added a commit that referenced this pull request Jul 22, 2024
@maxmwang
Revert "added local minio s3 and backup:mongodb script (#649)"
1f3829c 
This reverts commit bc514c0.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@kevinzwang kevinzwang kevinzwang approved these changes

@maxmwang maxmwang Awaiting requested review from maxmwang

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Write a script to backup to Google Cloud
2 participants
@ARtheboss @kevinzwang