-
Notifications
You must be signed in to change notification settings - Fork 1.2k
write_tsdb with metadata #1107
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
write_tsdb with metadata #1107
Conversation
Hello, Today's patch 32ce87f adds a new metadata tag :
This is useful for cases when pluginInstance (or typeInstance) may be set or not. Example :
In this example, when the Here are some opentsdb
Regards, |
I have compiled and verified that this PR works for me. It works well to convert information from the identifier to tags. I used the config below and I was able to get data from collectd to InfluxDB with host specific tags and good metric names to fit the InfluxDB tagging scheme. I think many others will find this to be useful as it is exactly what I expected to be able to do out of the box.
|
@ymettier I am getting tags without values with the above config for certain plugins. For example:
Any idea why the tag key is being inserted without the value? Thanks |
@ymettier I changed the code locally to check for an empty value as well as an empty key: In the define TSDB_STRING_APPEND_SPRINTF I added/modified these lines: const char *v = (value); \
if(k[0] != '\0' && v[0] != '\0') { \
n = ssnprintf(ptr, remaining_len, " %s=%s", k, v); \ This works for me and now tags are only added if they have a value. |
Thanks @nathanielc : I updated my code with your fix. See commit 87e08a2. |
Hello, InfluxDB >= 0.9 users, please read ! As described (shortly) in https://influxdb.com/docs/v0.9/write_protocols/opentsdb.html, InfluxDB accepts metrics encoded for OpenTSDB. So instead of sending "raw" Collectd metrics with the Collectd binary protocol, you can send them with write_tsdb. With this PR, you can easily rewrite the metrics names and add tags, exactly the same as it was planned for OpenTSDB. In other words, write_tsdb and this PR work very well with InfluxDB >= 0.9 !!! Regards, |
+1 I have been running collectd patched with this PR and sending metrics to InfluxDB for a few weeks now. With the config I mentioned above I get a measurement per plugin and the rest of the metadata becomes tags. Its been really useful. |
Running into this issue with collectd -> opentsdb. Thanks for working on this. I'm going to pull it down and give it a try. |
+1, some idea when will be merged into Master? this be amazing feature. Thanks to all that works around this. |
Great job! Using it with colltectd/Graphite-API/Grafana/InfluxDB. Cheers, |
Hey guys, I've run into a problem loading my configuration. I've split it up in multiple files and now I'm getting a:
Any idea what this means? Cheers, |
Hi All, I has the same error that reported before @szop85 To check if I miss something, some can share his collect.conf file to use the same and see if I made some mistake? The version I'm using be: collectd 5.5.0.377.g5afcfb3 Downloaded from: thanks for any advice! |
Hey @cdgraff, can you verify if you have the syslog plugin enabled? Cheers, |
Hi @szop85 no I don't has enabled, just in case i added and is the same error, just now this log into message file and I don't see in the console, but the message is the same. I'm using the example provided by @nathanielc... if be working for you, can share in some Gist the config file? thanks! |
After splitting my collectd config I forgot to activate/load the syslog plugin. After loading the syslog plugin, everything was fine. Cheers, |
@szop85 : this is a feature of my patch. If you do not patch Collectd with PR #1107, Collectd cannot understand Check if you successfully patched your Collectd and if you are really running the patched Collectd. Regards, |
@ymettier can show me a quick how to patch the master branch? I do this and got and error: 1107.patch:915: trailing whitespace. 1107.patch:1040: trailing whitespace. error: patch failed: src/collectd.conf.pod:8676 I'm not good with git... |
use "git merge" and merge my branch into Collectd master branch. |
I'm also testing this pull on my system for the last week and it is working very good. I would love to see this merged to master and in the next release, so I can move my systems to InfluxDB |
Functionally this LGTM |
when will this get merged? it's boring to patch and build collectd myself :-( @ymettier could you rebase your patch series against branch master or the latest tag "collectd-5.5.1"? Here is how I do it, just a simple conflict in src/write_tsdb.c:
This makes it much easier to merge than your original branch which involves many other upstream commits. The small conflict needs a one-line patch against collectd-5.5.1 + pull/1107:
If rebase against branch "master", there is also a small conflict,
This should be resolved by choosing the bottom half. |
|
||
if (vl->meta) { | ||
TSDB_META_DATA_GET_STRING(meta_tag[TSDB_TAG_PLUGIN]); | ||
if(temp) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Space after if
Review fixes in #1655 |
Any ETA on this? We would like to see this feature as well |
Superseeded by #1655 |
@ymettier My /etc/collectd.conf LoadPlugin match_regex And systemctl restart collectd thank you very much! |
Hello,
First of all, PR #1106 is a pre-requisite for this PR. This is why I based my work on that branch.
This patch allows use metadata to completely rewrite some metrics for OpenTSDB.
The problem is best explained in issue #709 and #887 : some metric names should not include the instance in their name but in a tag. For example,
cpu.1.cpu.user
should becomecpu.user
with an additional tagcpu=1
This patch is a proposition of a solution for this problem.
I added some metadata :
tsdb_tag_pluginInstance
tsdb_tag_type
tsdb_tag_typeInstance
tsdb_tag_dsname
If it is not empty, it will be the key of an opentsdb tag (the value is the item itself)
If it is empty, no tag is defined.
Here is how I use them for commonly used plugins cpu, df, disk, interface, load and swap :
Of course, when you do not define metadata, it works just as before.
Regards,
Yves