Add client side aggregation #844
Merged
+381
−110
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* add aggregator WIP * add metric types enum * add threading to metrics.py, add unsafe_flush * fix aggregator * add tests * remove print statements * remove lock for flush unsafe function * fix removing wrong lock * remove unecessary If * Run tests on any branch * Change MetricAggregator propreties so that it has all necessary fields to send to client * change aggregator to use MetricAggregator type * fix test * fix lint * remove enums * fix lint x2 * fix lint x3 * use existing threading function * remove import * fix lint * refactor _stop_flush_thread * rename variable * remove client from aggregator * move changes to base.py to next PR --------- Co-authored-by: Brian Floersch <[email protected]>
…g or aggregation before returning
skarimo
previously approved these changes
Aug 9, 2024
andrewqian2001datadog
dismissed stale reviews from skarimo and rayz
via
a784757
skarimo
previously approved these changes
Aug 9, 2024
rayz
previously approved these changes
Aug 12, 2024
|
/merge |
|
🚂 MergeQueue: pull request added to the queue The median merge time in Use |
andrewqian2001datadog
dismissed stale reviews from skarimo and rayz
via
5857802
rayz
approved these changes
Aug 12, 2024
andrewqian2001datadog
added a commit
that referenced
this pull request
Aug 12, 2024
This reverts commit 74ffb61.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Requirements for Contributing to this repository
What does this PR do?
Project description
This PR adds simple client side aggregation (aggregating metrics before they are received by the agent) for the python client. This PR combines two previous PRs together (1, 2)
Client side aggregation has already been implemented in Go, Java, and the C# dogstatsd clients
Description of the Change
Periodically aggregates metrics (Count, Gauge, Set) for a given
statsd_aggregation_flush_intervaland then sends the aggregated metrics to intake. A metric will be aggregated ifstatsd_disable_aggregatingisFalseand there was another matching context sent within the same flush interval.Verification Process
Unit tests that verify the aggregator.py class matches the existing behavior in the client side aggregator for the go client.
Unit tests that verify the new metric objects (merics.py) match the existing behavior in the client side aggregator for Go.
Local testing:
Confluence doc
More Docs
Bench Mark tests (TODO as a stretch goal)
Additional Notes
We currently do not allow buffering and aggregation at the same time, if both enabled, aggregation will occur instead of buffering.
Release Notes
Review checklist (to be filled by reviewers)
changelog/label attached. If applicable it should have thebackward-incompatiblelabel attached.do-not-merge/label attached.kind/andseverity/labels attached at least.