[CONFIG-325] Increase ldb download speeds #117
Conversation
| @@ -5,12 +5,17 @@ go 1.20 | |||
| require ( | |||
| github.com/AlekSi/pointer v1.0.0 | |||
| github.com/aws/aws-sdk-go v1.37.8 | |||
There was a problem hiding this comment.
can we free ourselves of all aws-sdk-go v1?
There was a problem hiding this comment.
I tried getting rid of awserr lib but there is really no equivalence in sdk v2, the most closed one would be smithy-go, but that would still require us making some minor logic changes in the testing file. I can definitely give it another try and you can take a look
There was a problem hiding this comment.
Okay so I freed reflector pkg from v1, but I then realized that there are much more places using v1, e.g., ledger_monitor which uses ecs client. I don't think it's gonna be a trivia task to completely free this repo from v1, would you think that's acceptable?
| } | ||
| } | ||
| n, err = io.Copy(w, reader) | ||
|
|
||
| events.Log("LDB inflated %d -> %d bytes", numBytes, n) |
There was a problem hiding this comment.
n is going to be 0 here if the file isn't gzipped
|
|
||
| events.Log("LDB inflated %d -> %d bytes", numBytes, n) | ||
|
|
||
| return |
There was a problem hiding this comment.
if the file isn't gzipped, we want n=numBytes
| Bucket: aws.String(d.Bucket), | ||
| Key: aws.String(d.Key), | ||
| }) | ||
| stats.Observe("snapshot_download_time", time.Now().Sub(start)) |
There was a problem hiding this comment.
Do you know the default string output of time.Now().Sub(start)? If it's ms then should you change the title of the metric to include _ms or whichever unit it is?
| return | ||
| } | ||
|
|
||
| func Unzip(src io.Reader, dest io.Writer) (int64, error) { |
There was a problem hiding this comment.
Do you want to add a metric around this start and end time as well in case? I'm not sure if it would only be in the io.Copy part or not.
zhou-hongyu
force-pushed
the
hzhou/CONFIG-325-increase-ldb-download-speeds
branch
from
2545d47
to
04be134
Compare
zhou-hongyu
force-pushed
the
hzhou/CONFIG-325-increase-ldb-download-speeds
branch
from
39c1e03
to
c410558
Compare
|
I know I may have asked this in our last 1-on-1, but do you have any benchmark / at least a snapshot of time of before / after this change? Even if you didn't profile the whole thing? If not should a separate PR be sent out first with just the metrics and then the change so the rough difference can be measured and observed? |
|
yes totally, the plan will be deploying to a ctlstore-beta node and validate the downloading speed via the dashboard I will circle back on this |
Sounds good |
…ing wal monitoring
zhou-hongyu
force-pushed
the
hzhou/CONFIG-325-increase-ldb-download-speeds
branch
from
c410558
to
467fb2e
Compare
aws-sdk-gowithaws-sdk-go-v2gzipwithklauspost/gizp