Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

hrpc: enable ScanMetrics #254

Merged
merged 1 commit into from
Jun 14, 2024
Merged

hrpc: enable ScanMetrics #254

merged 1 commit into from
Jun 14, 2024

Conversation

ciacono
Copy link
Collaborator

@ciacono ciacono commented Jun 4, 2024

Enable tracking scan metrics in the ScanResponse. Clients can access the metrics via calls to scanner.Next()
Support for HBase versions < 2.6.0 where ScanMetrics provides ROWS_SCANNED and ROWS_FILTERED metrics.

scanner.go Outdated Show resolved Hide resolved
integration_test.go Outdated Show resolved Hide resolved
scanner.go Outdated Show resolved Hide resolved
@ciacono ciacono force-pushed the enable-scanmetrics branch 2 times, most recently from 74ea11a to 1c3410a Compare June 4, 2024 20:41
Copy link
Collaborator

@dethi dethi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left a few comments but I also wonder if adding the scan metrics to the hrpc.Results is the right choice here. It seems that we will be allocating a lot of small struct on the heap to hold these result and most likely the user of this feature wants to know the overall scan metrics at the end so they will have to sum the various result in their own code.

Would it make more sense to hold the scan metrics in the scanner struct itself, accumulating the result as we fetch and add a method to the scanner to returns the end metrics. And also doing that while keepting the scan metrics as map[string]int64 such that there is no dependencies with the HBase version at all (if a new version of HBase introduce a new scan metrics name, or change one, we don't need to update GoHBase accordintly).

What are your thoughts on this?

pb/Client.proto Outdated Show resolved Hide resolved
scanner.go Outdated Show resolved Hide resolved
scanner.go Outdated Show resolved Hide resolved
scanner.go Outdated Show resolved Hide resolved
scanner.go Show resolved Hide resolved
@tsuna
Copy link
Owner

tsuna commented Jun 11, 2024

I echo Thibault's feedback here that always passing around the scan metrics along with the batch of results makes the code more complex while also making the scan metrics harder to use from the caller's point of view. Not sure we should keep them in a generic map[string]int64 as this would require assuming we can always accumulate all the metrics by summing them up, but either way I do agree we should track the metrics directly in the scan object instead of passing them around all the time separately, so that the caller can consult the cumulative metrics at the end of the scan, which will most likely be the most common use case.

@ciacono ciacono force-pushed the enable-scanmetrics branch 2 times, most recently from 46d7b0d to 643c294 Compare June 12, 2024 21:14
scanner.go Show resolved Hide resolved
scanner.go Outdated Show resolved Hide resolved
scanner.go Outdated Show resolved Hide resolved
Enable tracking scan metrics in the ScanResponse. Clients can access
the metrics via calls to scanner.GetScanMetrics()
Support for  HBase versions < 2.6.0 where ScanMetrics provides
ROWS_SCANNED and ROWS_FILTERED metrics.
Copy link
Collaborator

@dethi dethi left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM 🚢

@dethi dethi merged commit c8b69fd into tsuna:master Jun 14, 2024
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants