-
Notifications
You must be signed in to change notification settings - Fork 5.1k
Description
Greetings! Here's a feature request to track. Other folks that use Datadog for monitoring would likely be interested in this as well.
The statsd metrics that Envoy produces are geared towards the generic statsd implementation. For folks that use Datadog for monitoring this is suboptimal, because Datadog's statsd implementation supports metric tags (ie. dimensions). See:
- https://docs.datadoghq.com/guides/dogstatsd/
- https://www.datadoghq.com/blog/the-power-of-tagged-metrics/
- https://help.datadoghq.com/hc/en-us/articles/204312749-Getting-started-with-tags
Ex: For a given metric like a cluster's upstream_cx_rx_bytes_buffered
, a single Envoy instance currently produces metrics with names:
envoy.cluster.CLUSTER_A.upstream_cx_rx_bytes_buffered
envoy.cluster.CLUSTER_B.upstream_cx_rx_bytes_buffered
...
envoy.cluster.CLUSTER_N.upstream_cx_rx_bytes_buffered
... for each cluster. In Datadog best practices, one would prefer a single metric-name like envoy.cluster.upstream_cx_rx_bytes_buffered
which is tagged by the respective cluster name. This makes it possible to aggregate & monitor across many clusters, and be able to find anomalies or get alerts from any misbehaving cluster(s).
(And yes, part of the problem here is that Datadog doesn't support templating/string interpolation of metrics names in their monitors & dashboards.)
Activity
mattklein123 commentedon Aug 24, 2017
Tags/dimensions is something we would like to support. Also for Prometheus/Wavefront/Influx, etc. Needs some thinking on where to do this and how to keep back compat for statsd.
dnoe commentedon Aug 24, 2017
@mrice32 This might be relevant to your interests.
[-]Support for Datadog statsd Metric Tags[/-][+]Support for Statsd Tags/Dimensions[/+]tehranian commentedon Aug 24, 2017
Didn't know that Prometheus/Wavefront/Influx supported tags as well. Changed the title of this issue to be more generic.
Thanks folks!
mrice32 commentedon Aug 24, 2017
+1 We will definitely need this feature for our internal monitoring as well.
mattklein123 commentedon Aug 24, 2017
This one will need some design thought. When someone firmly signs up to work on it we can figure out what to do here. @mrice32 if that's you, let's chat at some point.
mrice32 commentedon Aug 24, 2017
Yes, that will be me. Sounds good.
mattklein123 commentedon Aug 24, 2017
OK ping me offline when you are ready to work on this and we can chat and hopefully come back to this thread with a design proposal. Anyone else feel free to weigh in here with thoughts.
mattklein123 commentedon Aug 28, 2017
I just synced up with @mrice32 and after some brainstorming this is what I'm thinking about at a high level for this:
I think this approach balances efficiency, user configuration, and backcompat.
Please comment!