Fix: add hard limit on notion db size allowed to be turned into CSV. ContinueAsNew each time #7444
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Fraggle
force-pushed
the
1337-add-hard-limit-for-notion-database-import-size
branch
2 times, most recently
from
a6772b2
to
af1dae9
Compare
…ContinueAsNew each time.
Fraggle
force-pushed
the
1337-add-hard-limit-for-notion-database-import-size
branch
from
af1dae9
to
bc15b7a
Compare
fontanierh
approved these changes
Sep 17, 2024
| const pagesProperties = pageCacheEntries.map( | ||
| (p) => JSON.parse(p.pagePropertiesText) as PageObjectProperties | ||
| ); | ||
| let pagesProperties: PageObjectProperties[] = []; |
There was a problem hiding this comment.
I think you could also avoid holding onto this entirely by generating smaller CSVs as you go through the chunks
There was a problem hiding this comment.
This is what I was going for initially but:
- there is some header management to do if I have several small csv
- in the end, the text would be turned into text (as csv) in memory anyway, I wouldn't be saving anything :(
| @@ -120,164 +114,152 @@ export async function notionSyncWorkflow({ | |||
|
|
|||
| const isInitialSync = !lastSyncedPeriodTs; | |||
|
|
|||
| do { | |||
Fraggle
deleted the
1337-add-hard-limit-for-notion-database-import-size
branch
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Some customers have very large notion DB. We now have a hard limit of 256mb to allow them to be turned into CSV.
Use raw data fetching by chunk of 250 items to limit memory usage.
Removed the 32 iterations before continuing as new to simplify.
Risk
Should be harmless.
Deploy Plan
Deploy
connectors