-
Notifications
You must be signed in to change notification settings - Fork 1.7k
Add support for Elasticsearch after 5.0(including new version 6.0.0) #1426
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
I tried this but the first time it threw an exception and the docker container exited with returncode <> 0 afterwards.
|
In my elastalert_status index I only have entries like these
|
@CodingSpiderFox in elasticsearch version 6.0 you can only have one type in mapping of new created index, if you have more than one doc_type, you will get the error 'Rejecting mapping update to [your_index] as the final mapping would have more than 1 type'; Unfortunately the project contains condition code and create 4 or more types to separate the data, such as This pull request only works with the old index you created before 6.0, and you can use the old index in version 6.0, too. reference: Removal of mapping types |
How can I use the old index? Do I have to start my elastic 5 stack, then create the old index with an old version of elastalert, then upgrade to elastic 6 and then start this new version of elastalert? |
@CodingSpiderFox yes, you are right. |
How can I upgrade from ES5 to ES6? Is there a proposed upgrade procedure by the developers of the ELK stack? |
@CodingSpiderFox It's not the main topic here, please have a look at ELK official website. |
ES6 only allows 1 mapping type per index. My suggestion would be to create one index for each doctype and instead of referring to the same writeback_index, refer to the doctype index. Only requires a few code changes. Datatype 'string' does not exist anymore, use 'keyword' instead and index: 'not_analyzed' can be removed when you define your string as 'keyword'. Got it to work on this pull request with the mentioned changes. |
The I had to re-run before the setup worked with ES6 I had to pip install elasticsearch==6.0.0. |
Is anyone working on native elastic 6 support already? @Jensvandecasteele are you planning to make a pull request with the changes you made? @danielpops: In case no one is already working on full elastic 6 support, I could try to do that. But I need to be sure that if I do a pull request with working elastic 6 support, it will be merged in an appropriate timeframe. When I look at the open pull requests, I get the impression that your organization does not care that much about contributors spending their time to improve elastalert. There are several open PRs which didn't even receive a comment by a maintainer of yelp/elastalert (for example #867 #834). I think this is not a good way to honour the time people spend on doing development here. To close an issue or a PR without solving/merging it could also be an acceptable action sometimes like for example if an issue is rather a question on how to use elastalert (could be discussed answered in a forum, stackoverflow, etc. if you want the users to do it that way). As long as you state the reason, this would be better than not even answering. |
I started out with a clean ES6 stack. My fixes are a little quick and dirty at some points, but at least it's working. I will have a look this weekend to put the code online. |
@Jensvandecasteele great, thanks! |
Code has been checked in (pull request on Lunatictwo's fork). It's definitely not a final solution, but it gets you going if you just want to try it out on a clean ES6 stack. |
Will try it out tomorrow if I find the time |
quick&dirty fix to install on clean ES6
It works with elastic stack 6.0.0-beta2 (I used beta2 because I had other problems with Kibana and I thought they weren't there in 6.0.0-beta2 .. now testing 6.0.0 stable) |
Works with 6.0.0, too but now the ticket seems to be created for every match of the rule:
Obviously, the issues only differ in the microseconds field:
a) Is that a known issue? |
A) Maybe, I only started digging into Elastalert last week. I did however come across the same phenomenon. When I was using a term query I received 1 alert per match, when I used a query string I only received 1. Try using a query string instead. The reason why I changed was because multiple terms caused query issues. B) It's possible, although I did not change anything about that part as far as I know. |
@Jensvandecasteele yes, maybe that's what my colleague tried to tell me. Something with the queries has changed and needs to be adapted, too, he said. |
@CodingSpiderFox I have been super bad about responding to the pull requests, sorry about that. I've not completely forgotten about them though and I've been meaning to go through all the pending ones and either comment/accept/close. I've introduced a number of bugs by not thoroughly going through them, so I'd like to be careful. We don't use ES 6, Kibana 5, and aren't actively adding any additional features right now, so it's unfortunately taken a back burner for the time being. What I need to do (and should have done a long time ago) is create a development branch and ask people to merge into that instead of master, so I can stage changes together more easily without worrying about breaking things for anyone who clones from master instead of pulling a release. Thank you all for the contributions, I know it's frustrating to have them sit unmerged, I'll take responsibility for that. |
@Qmando thanks for your effort and undestanding. I think creating a development branch is a good idea :) |
@Jensvandecasteele: The same issue still occurs with with the query_string. My rule looks like this now:
The result still looks like this: Elastalert gives the following error message:
Could it be that this is a type error in the python source code? Looks like the String is not handeled correctly because the |
@Qmando what about having a quick workaround in terms of enforcing "is_five" through configuration file? In our case we don't do much of filtering, but rely on documents count, so just replacing "filtered" with "bool" should be sufficient to pass API validation until proper support for ES 6.x is implemented. |
also, it is worth pointing out that in my case the indexes were not created automatically, so I had to go into |
It's very good. It solves my problem. Except for this type of rule:doc_type must be specified. |
Is there somewhere a list of the pending blocking "stuff" to follow for the release ? |
Thank you @Lunatictwo and others for your work here, it's greatly appreciated, I've merged in #1472 which was based off of these changes. |
I am having issue when running the elastalert in docker container against ES6.1.2 any work around for this
05:42:29.159Z INFO elastalert-server: Config: No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json. |
I am using it against 6.1 and 6.2 without any problem. I wrote a small DIY in my blog on how I dockerized ElastAlert here: https://mannekentech.com/2018/01/19/using-elastalert/ Cheers, Arnaud |
@phoenixjun
There are too many things in the bitsensor version. We are integrating our own elast alert configuration system, so we only want the elast alert kernel not the Rest API on top of it.
Cheers,
… On 15 Mar 2018, at 13:01, phoenixjun ***@***.***> wrote:
@snuids <https://github.com/snuids>
I was using the container from the bitsensor repo from the home page of this repository.
docker run -d -p 3030:3030
-v pwd/config/elastalert.yaml:/opt/elastalert/config.yaml
-v pwd/config/config.json:/opt/elastalert-server/config/config.json
-v pwd/rules:/opt/elastalert/rules
-v pwd/rule_templates:/opt/elastalert/rule_templates
--net="host"
--name elastalert bitsensor/elastalert:latest
Why do you want to build the image yourself?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#1426 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AMkJQ6yw3kLGVL8E77vC95JtPuZTyvNiks5telf9gaJpZM4Qf9_P>.
|
@phoenixjun |
@snuids I see you blog and follow it. my elasticsearch version is 6.2.4, but when I run create_index.py and enter string but it error. below is my enter and output, do you know why? Thanks.
|
It looks like a problem in your elastic search installation. The command es.info() performs an http get on the address http://localhost:9200. If you load this address in your browser, you should receive a json as answer. If you cannot access this address via your browser, it cannot work via the create_index command. Cheers, |
thank you for your reply. Thanks. |
@cherubic Don't enter anything for I'll edit this to be more clear. I think it's pretty uncommon that people use this. |
Note that I am also having the same issues as @cherubic and am not entering anything in the prefix. A small self written script also confirms the the elasticsearch python modules works for es.info().
|
Elasticsearch 6.0.0 was Released.
Changed the way of judging version of es, in order to support version after 5.x including 6.x.