What a generic interface? to add and set arbitrary metadata

<Chain "PreCache">
  <Rule "add_sys_prefix">
    <Match "regex">
      Host "^"
    </Match>
    <Target "set">
      MetaDataSet "tsdb_prefix" "sys."
      MetaDataAdd "tsdb_tags" "status=production"
    </Target>
  </Rule>
</Chain>

Perhaps MetaDataAdd needs one more argument to set the concatenation string.

+1, I'd also prefer a more generic "set metadata" functionality.

@octo okay I'll see what I can come up with. What do you think about the interface for the concat problem @manuelluis pointed out?

In the Set target, I'd probably only go with a Metadata option which (re)sets the given metadata.

If you need more complex manipulation of metadata, I'd go for a separate Metadata target, i.e. copy and adapt the Set target to handle whatever metadata manipulation you need. Keep in mind that other people may like to extend this, but I wouldn't take this to eleven and implement all manipulation you can come up with. If people need to toggle a boolean, for example, let them send a feature request or PR for this before making the new code unnecessarily complex.

I personally would prefer a different Set target for Prefix versus Tags.

Prefix is definitely something I can see being useful in rules, but I'd personally keep the Tags to something host-specific.
I know that others might do the opposite, modify both with a Rule(like the example above), and/or completely avoid Prefix (or Tags - but doubtful) altogether.

So having these separate makes this much easer to work with from a configuration management perspective.

Hello,

I'm adressing here this issue and the issue #887.

I think the real need for opentsdb write plugin is to redefine the UID and define tags.

Define the UID

• add a prefix
• in some case (including the df or cpu from the example of issue write_tsdb metric names and tags #887), remove PluginInstance and/or TypeInstance

Add tags

• add some static tags (like "status=production" here)
• add some tags with values from PluginInstance and/or TypeInstance (like in issue write_tsdb metric names and tags #887)

Example/Suggestion :

<Chain "PreCache">
  <Rule "opentsdb_df">
    <Match "regex">
      Plugin "^df$"
    </Match>
    <Target "set">
      MetaDataSet "tsdb_id" "sys.df.%Type%.%TypeInstance%"
      MetaDataAdd "tsdb_tags" "mount=%PluginInstance%"
    </Target>
  </Rule>

  <Rule "opentsdb_cpu">
    <Match "regex">
      Plugin "^cpu$"
    </Match>
    <Target "set">
      MetaDataSet "tsdb_id" "sys.cpu.%Type%"
      MetaDataAdd "tsdb_tags" "cpu=%PluginInstance%"
    </Target>
  </Rule>

  <Rule "add_sys_prefix">
    <Match "regex">
      Host "^"
    </Match>
    <Target "set">
      MetaDataSet "tsdb_prefix" "sys."
      MetaDataAdd "tsdb_tags" "status=production"
    </Target>
  </Rule>
</Chain>

Note : this merges the need to add "sys." as a prefix and the 2 examples from issue #887 :

Having

put df.df_complex.reserved 1420724763 12345678 fqdn=machine.domain.tld,mount=boot
put cpu.idle 1420724763 12345678 fqdn=machine.domain.tld,cpu=0

instead of

put df.boot.df_complex.reserved 1420724763 12345678 fqdn=machine.domain.tld
put cpu.0.cpu.idle 1420724763 12345678 fqdn=machine.domain.tld

Explanation

1/ add a tsdb_id metadata.

In the write_tsdb plugin, if the tsdb_id exists, use it and ignore p.pi.t.ti.dsname

Also ignore any tsdb_prefix ? I'm not sure. If you take my example, yes, ignore it, because it is already in the tsdb_id. I'm not sure because it may be a good idea to have a generic prefix.

If the tsdb_id does not exist, create it from p.pi.t.ti as it is already done in write_tsdb.c.

2/ add tags with values from pluginInstance and/or typeInstance

As show in the example, we need 3 variables :

• %pluginInstance%
• %typeInstance%
• %dsname%
  I don't think we need other variables

The metadata tags will take their values from the pi or ti or dsname from the metric.

What do you think ?

Regards,
Yves

Hello,

I wrote a patch in order to implement MetaDataSet. Please see PR #1106.

While I was writing this patch, I was thinking on how to use it here.
First thing : using variables (like %pluginInstance%) will make the code a lot more complex. This is a bad idea.

Here is my new suggestion/example :

<Chain "PreCache">
  <Rule "opentsdb_df">
    <Match "regex">
      Plugin "^df$"
    </Match>
    <Target "set">
# put df.df_complex.reserved 1420724763 12345678 fqdn=machine.domain.tld,mount=boot
      MetaDataSet "tsdb_prefix" "sys."
      MetaDataSet "tsdb_tag_pluginInstance" "mount"
    </Target>
  </Rule>

  <Rule "opentsdb_cpu">
    <Match "regex">
      Plugin "^cpu$"
    </Match>
    <Target "set">
# put sys.cpu.idle 1420724763 12345678 fqdn=machine.domain.tld,cpu=0
      MetaDataSet "tsdb_prefix" "sys."
      MetaDataSet "tsdb_tag_pluginInstance" "cpu"
      MetaDataSet "tsdb_tag_type" ""
    </Target>
  </Rule>

  <Rule "add_sys_prefix">
    <Match "regex">
      Host "^"
    </Match>
    <Target "set">
      MetaDataSet "tsdb_prefix" "sys."
      MetaDataAdd "tsdb_tags" "status=production"
    </Target>
  </Rule>
</Chain>

The write_tsdb plugin would work like this :

• If the tsdb_prefix is defined, add it.
• if tsdb_tag_pluginInstance is defined and is empty, remove pluginInstance from the ID
• if tsdb_tag_pluginInstance is defined and is not empty, remove pluginInstance from the ID, but add it as a tag
• Same for tsdb_tag_type
• Same for tsdb_tag_typeInstance
• Same for tsdb_tag_dsname

Not in this example, but another meta data could be tsdb_id in order to force the id (but don't use variables as I suggested in my previous post).

What do you think ?

Regards,
Yves

Hello,

Please check if PR #1107 can be a solution.

Regards,
Yves

Any ETA on getting a solution to the TSDB issue resolved? Is the Pull Request satisfactory?

I think this pull request is superseded by pull request #1107.