Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: make kurtosis service logs faster #2525

Merged
merged 25 commits into from
Aug 12, 2024
Merged

feat: make kurtosis service logs faster #2525

merged 25 commits into from
Aug 12, 2024

Conversation

tedim52
Copy link
Contributor

@tedim52 tedim52 commented Aug 9, 2024

Description

Users were experiencing kurtosis service logs taking a long time. After running tests, I discovered that a majority of execution time during log processing was spent in the following lines:

logLines := []logline.LogLine{*logLine}
userServicesLogLinesMap := map[service.ServiceUUID][]logline.LogLine{
      serviceUuid: logLines,
}
logsByKurtosisUserServiceUuidChan <- userServicesLogLinesMap

Prior to this change, we were sending logs one at a time on an unbuffered channel - unbuffered channels block until the receiving goroutine reads the value. This was causing a lot of time being wasted waiting to send log lines across the channel.

This change implements a LogLineSender that:

  1. uses a buffered go channel (won't block on sending line unless buffer is full)
  2. batches log lines (reduces read overhead, receiving goroutine performs fewer reads/sends)

With this change, the time to read 20 minutes of cl-lighthouse-geth logs with log level set to debug went from 1min53sec to 30.055 seconds. The time to read 2 hours 10 minutes worth of cl-lighthouse debug logs (around 3.4 gb of logs) went from 15min1sec to 3min31 sec. (As a benchmark, cat logs.json on 3.4 gb of logs takes around 2min - on my machine - so much closer) This can likely be improved further by tuning the buffer size and batch amount.

Is this change user facing?

YES

References:

https://discord.com/channels/783719264308953108/1267837033032974467/1267842228072611881

.circleci/config.yml Outdated Show resolved Hide resolved
@tedim52 tedim52 enabled auto-merge August 12, 2024 15:17
@tedim52 tedim52 added this pull request to the merge queue Aug 12, 2024
Merged via the queue into main with commit d6b246a Aug 12, 2024
51 checks passed
@tedim52 tedim52 deleted the tedi/logspeedup branch August 12, 2024 15:39
github-merge-queue bot pushed a commit that referenced this pull request Aug 16, 2024
🤖 I have created a release *beep* *boop*
---


##
[1.1.0](1.0.0...1.1.0)
(2024-08-16)


### Features

* [log retention improvements pt. 1] introduce file layout interface
([#2534](#2534))
([a494278](a494278))
* make kurtosis service logs faster
([#2525](#2525))
([d6b246a](d6b246a))

---
This PR was generated with [Release
Please](https://github.com/googleapis/release-please). See
[documentation](https://github.com/googleapis/release-please#release-please).

---------

Co-authored-by: kurtosisbot <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants