Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Mensie logs #123

Open
TomiBelan opened this issue May 8, 2018 · 0 comments
Open

Mensie logs #123

TomiBelan opened this issue May 8, 2018 · 0 comments

Comments

@TomiBelan
Copy link
Member

logs a oldlogs zaberaju privela miesta, lebo pocet pouzivatelov Votru rastie rychlejsie, ako sme cakali.

Casom asi bude treba vacsie disky, ale aj dovtedy je co optimalizovat.

  • logs obsahuje nekomprimovane logy za posledne 3-4 dni. Treba komprimovat. Asi radsej gz, kvoli rychlosti a pamati. (Kompresia bude per-request namiesto per-session, ale to je snad OK.)
  • oldlogs obsahuje gz logy jednotlivych sessionov. Chceme prejst na tar.xz logy vela sessionov, lebo xz je podstatne lepsi ako gz a tar.xz je podstatne lepsi ako vela jednotlivych xz suborov.
  • S tar.xz sa bude velmi tazko pracovat - uz nebude fungovat zgrep "nejaky pattern" $(nejaka mnozina suborov). Treba do logutil pridat nove nastroje, ktore to zjednodusia, a otestovat ze su dostatocne rychle.
TomiBelan added a commit that referenced this issue May 8, 2018
@TomiBelan
Write gzip compressed logs.
6ad1e31 
Per-RPC compression is not as good as per-file compression, but the size
difference is small (+3.8% in a test from last January).

Cron support is not implemented yet. Logutil support is partial.

For #123.
TomiBelan added a commit that referenced this issue May 8, 2018
@TomiBelan
Decrease default log_max_age.
d0bb779 
For #123.
TomiBelan added a commit that referenced this issue Dec 16, 2018
@TomiBelan
Compress logs as .tar.xz files.
d9bf277 
Logs are now stored in:

1. logs/YYYYMMDD_*.gz (while still active)
2. oldlogs/YYYYMMDD_*.xz (when the session expires)
3. logarchive/YYYYMM.tar.xz (created after each month)

Experiments on old logs show that level 2 is 3x smaller than level 1, and
level 3 is 13x smaller than level 1.

logutil.py doesn't support reading level 3 logs yet.

For #123.
@TomiBelan TomiBelan mentioned this issue Dec 16, 2018
Merged
TomiBelan added a commit that referenced this issue Dec 22, 2018
@TomiBelan
Compress logs as .tar.xz files.
780b21c 
Logs are now stored in:

1. logs/YYYYMMDD_*.gz (while still active)
2. oldlogs/YYYYMMDD_*.xz (when the session expires)
3. logarchive/YYYYMM.tar.xz (created after each month)

Experiments on old logs show that level 2 is 3x smaller than level 1, and
level 3 is 13x smaller than level 1.

logutil.py doesn't support reading level 3 logs yet.

For #123.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@TomiBelan