Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

RDBC-864 Node.js client bulk insert performance #442

Merged
merged 4 commits into from
Aug 5, 2024
Merged

Conversation

ml054
Copy link
Member

@ml054 ml054 commented Aug 2, 2024

@ml054 ml054 added the WIP label Aug 2, 2024
public async flushIfNeeded(force = false): Promise<void> {
if (this._currentWriter.length > this._maxSizeInBuffer || this._asyncWriteDone) {
if (this.isFlushNeeded() || this._asyncWriteDone) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that this._asyncWriteDone should be in isFlushNeeded

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes. It generates more falses in tryStoreSync, but we wait less overall.

id: string,
metadata?: IMetadataDictionary): boolean {

if (this.isFlashNeeded() || this._first) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

isFlushNeeded, no?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no, first guarantees that stream was ensured and it was initialized.

@@ -851,6 +878,10 @@ export class BulkInsertOperation extends BulkInsertOperationBase<object> {
}
}

private isFlashNeeded(): boolean {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
private isFlashNeeded(): boolean {
private isFlushNeeded(): boolean {

@ml054 ml054 removed the WIP label Aug 3, 2024
@ayende
Copy link
Member

ayende commented Aug 3, 2024

Trying to run with this gives me:

  [cause]: ReferenceError: fetch is not defined
      at eu.send (C:\Work\tmp\node_modules\ravendb\dist\cjs\ravendb.cjs:5:4667)
      at o._send (C:\Work\tmp\node_modules\ravendb\dist\cjs\ravendb.cjs:5:44540)
      at async o._sendRequestToServer (C:\Work\tmp\node_modules\ravendb\dist\cjs\ravendb.cjs:5:43256)
      at async o._executeOnSpecificNode (C:\Work\tmp\node_modules\ravendb\dist\cjs\ravendb.cjs:5:41974)
      at async o._handleServerDown (C:\Work\tmp\node_modules\ravendb\dist\cjs\ravendb.cjs:5:51434)
      at async o._sendRequestToServer (C:\Work\tmp\node_modules\ravendb\dist\cjs\ravendb.cjs:5:44316)
      at async o._executeOnSpecificNode (C:\Work\tmp\node_modules\ravendb\dist\cjs\ravendb.cjs:5:41974)
      at async bs._getNextRange (C:\Work\tmp\node_modules\ravendb\dist\cjs\ravendb.cjs:10:28284)
      at async bs.getNextId (C:\Work\tmp\node_modules\ravendb\dist\cjs\ravendb.cjs:10:28051)
      at async bs.generateDocumentId (C:\Work\tmp\node_modules\ravendb\dist\cjs\ravendb.cjs:10:27699)
  • checked out v6.0
  • pulled your changes
  • npm run build, npm pack
  • npm i /path/to/ravendb-6.0.0.tgz
 const store = new DocumentStore('http://localhost:8080', 'bulk');
    store.initialize();
    const bulk = store.bulkInsert();
    for (let i = 0; i < 100_000_000; i++) {
        const user = new User('user' + i);
        const id = "users/" + i;
        if (bulk.tryStoreSync(user, id) == false) {
            await bulk.store(user, id);
        }
    }
    console.log('done storing');
    await bulk.finish();

Tried with and without tryStoreAsync, same error.

@ayende
Copy link
Member

ayende commented Aug 3, 2024

When running in debug, it complains about missing ReadableStream and then about fetch.

$ node --version
v16.14.0

Upgraded to node v20 and it appears to be working

@ml054
Copy link
Member Author

ml054 commented Aug 4, 2024

in 6.0 we require node.js 18+
https://github.com/ravendb/ravendb-nodejs-client/blob/v6.0/package.json#L40

however node 16 doesn't seem to respect this setting. :(

@ml054 ml054 merged commit f096d6e into ravendb:v6.0 Aug 5, 2024
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants