Skip to content

Support for Statsd Tags/Dimensions #1536

@tehranian

Description

@tehranian
Contributor

Greetings! Here's a feature request to track. Other folks that use Datadog for monitoring would likely be interested in this as well.

The statsd metrics that Envoy produces are geared towards the generic statsd implementation. For folks that use Datadog for monitoring this is suboptimal, because Datadog's statsd implementation supports metric tags (ie. dimensions). See:

Ex: For a given metric like a cluster's upstream_cx_rx_bytes_buffered, a single Envoy instance currently produces metrics with names:

envoy.cluster.CLUSTER_A.upstream_cx_rx_bytes_buffered
envoy.cluster.CLUSTER_B.upstream_cx_rx_bytes_buffered
...
envoy.cluster.CLUSTER_N.upstream_cx_rx_bytes_buffered

... for each cluster. In Datadog best practices, one would prefer a single metric-name like envoy.cluster.upstream_cx_rx_bytes_buffered which is tagged by the respective cluster name. This makes it possible to aggregate & monitor across many clusters, and be able to find anomalies or get alerts from any misbehaving cluster(s).

(And yes, part of the problem here is that Datadog doesn't support templating/string interpolation of metrics names in their monitors & dashboards.)

Activity

mattklein123

mattklein123 commented on Aug 24, 2017

@mattklein123
Member

Tags/dimensions is something we would like to support. Also for Prometheus/Wavefront/Influx, etc. Needs some thinking on where to do this and how to keep back compat for statsd.

dnoe

dnoe commented on Aug 24, 2017

@dnoe
Contributor

@mrice32 This might be relevant to your interests.

changed the title [-]Support for Datadog statsd Metric Tags[/-] [+]Support for Statsd Tags/Dimensions[/+] on Aug 24, 2017
tehranian

tehranian commented on Aug 24, 2017

@tehranian
ContributorAuthor

Didn't know that Prometheus/Wavefront/Influx supported tags as well. Changed the title of this issue to be more generic.

Thanks folks!

mrice32

mrice32 commented on Aug 24, 2017

@mrice32
Member

+1 We will definitely need this feature for our internal monitoring as well.

mattklein123

mattklein123 commented on Aug 24, 2017

@mattklein123
Member

This one will need some design thought. When someone firmly signs up to work on it we can figure out what to do here. @mrice32 if that's you, let's chat at some point.

mrice32

mrice32 commented on Aug 24, 2017

@mrice32
Member

Yes, that will be me. Sounds good.

mattklein123

mattklein123 commented on Aug 24, 2017

@mattklein123
Member

OK ping me offline when you are ready to work on this and we can chat and hopefully come back to this thread with a design proposal. Anyone else feel free to weigh in here with thoughts.

mattklein123

mattklein123 commented on Aug 28, 2017

@mattklein123
Member

I just synced up with @mrice32 and after some brainstorming this is what I'm thinking about at a high level for this:

  • We will always need to retain backcompat for stat backends that don't supporting tagging.
  • Instead of having tagging/not-tagging be built throughout the code, it would be better if tagging is orthogonal to the main stats.
  • We propose adding the ability to specify tag extraction via effectively regex capture expressions. For example cluster..foo.bar.baz would capture the cluster name as a tag.
  • Because of how heavily cached stats are, we can do tag extraction when stats are first created (for some of the dynamic stats we might want to have them pre-cached on the scope also).
  • Then when stats are sent to sinks, sinks will have access to full hierarchical name, as well as tag extracted name.

I think this approach balances efficiency, user configuration, and backcompat.

Please comment!

modified the milestone: 1.5.0 on Aug 30, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementFeature requests. Not bugs or questions.

Type

No type

Projects

No projects

Relationships

None yet

    Development

    No branches or pull requests

      Participants

      @tehranian@dnoe@mattklein123@mrice32

      Issue actions

        Support for Statsd Tags/Dimensions · Issue #1536 · envoyproxy/envoy