Skip to content

Add support for Elasticsearch after 5.0(including new version 6.0.0) #1426

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 9 commits into from

Conversation

emhlbmc
Copy link

@emhlbmc emhlbmc commented Nov 16, 2017

Elasticsearch 6.0.0 was Released.
Changed the way of judging version of es, in order to support version after 5.x including 6.x.

@spotlesscoder
Copy link
Contributor

I tried this but the first time it threw an exception and the docker container exited with returncode <> 0 afterwards.
I started it again. Now it repeats showing exceptions like this one:

elastalert_1  | 2017-11-16 17:41:39,334 DEBG 'elastalert' stderr output:
elastalert_1  | INFO:elastalert:Queried rule Many Logs Mail Alert from 2017-11-16 17:41 UTC to 20
11-16 17:41 UTC: 480 hits
elastalert_1  |
elastalert_1  | 2017-11-16 17:41:39,338 DEBG 'elastalert' stderr output:
elastalert_1  | WARNING:elasticsearch:POST http://elastic:9200/elastalert_status/elastalert
atus [status:400 request:0.003s]
elastalert_1  |
elastalert_1  | 2017-11-16 17:41:39,338 DEBG 'elastalert' stderr output:
elastalert_1  | ERROR:root:Error writing alert info to Elasticsearch: TransportError(400, u'illeg
argument_exception', u'Rejecting mapping update to [elastalert_status] as the final mapping would
ve more than 1 type: [elastalert_error, elastalert_status]')
elastalert_1  | Traceback (most recent call last):
elastalert_1  |   File "/opt/elastalert/elastalert/elastalert.py", line 1389, in writeback
elastalert_1  |     doc_type=doc_type, body=body)
elastalert_1  |   File "build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py", line 76, in
apped
elastalert_1  |     return func(*args, params=params, **kwargs)
elastalert_1  |   File "build/bdist.linux-x86_64/egg/elasticsearch/client/__init__.py", line 300,
 index
elastalert_1  |     _make_path(index, doc_type, id), params=params, body=body)
elastalert_1  |   File "build/bdist.linux-x86_64/egg/elasticsearch/transport.py", line 314, in pe
rm_request
elastalert_1  |     status, headers, data = connection.perform_request(method, url, params, body,
aders=headers, ignore=ignore, timeout=timeout)
elastalert_1  |   File "build/bdist.linux-x86_64/egg/elasticsearch/connection/http_requests.py",
e 90, in perform_request
elastalert_1  |     self._raise_error(response.status_code, raw_data)
elastalert_1  |   File "build/bdist.linux-x86_64/egg/elasticsearch/connection/base.py", line 125,
 _raise_error
elastalert_1  |     raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_mes
e, additional_info)
elastalert_1  | RequestError: TransportError(400, u'illegal_argument_exception', u'Rejecting mapp
 update to [elastalert_status] as the final mapping would have more than 1 type: [elastalert_erro
elastalert_status]')

@spotlesscoder
Copy link
Contributor

In my elastalert_status index I only have entries like these

{
  "_index": "elastalert_status",
  "_type": "elastalert_error",
  "_id": "YXvqxV8BX-Qf7TvaohIE",
  "_version": 1,
  "_score": null,
  "_source": {
    "message": "Error while querying for alert silence status: TransportError(400, u'search_phase_execution_exception', u'No mapping found for [until] in order to sort on')",
    "traceback": [
      "Traceback (most recent call last):",
      "  File \"/opt/elastalert/elastalert/elastalert.py\", line 1667, in is_silenced",
      "    size=1, body=query, _source_include=['until', 'exponent'])",
      "  File \"build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py\", line 76, in _wrapped",
      "    return func(*args, params=params, **kwargs)",
      "  File \"build/bdist.linux-x86_64/egg/elasticsearch/client/__init__.py\", line 636, in search",
      "    doc_type, '_search'), params=params, body=body)",
      "  File \"build/bdist.linux-x86_64/egg/elasticsearch/transport.py\", line 314, in perform_request",
      "    status, headers, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)",
      "  File \"build/bdist.linux-x86_64/egg/elasticsearch/connection/http_requests.py\", line 90, in perform_request",
      "    self._raise_error(response.status_code, raw_data)",
      "  File \"build/bdist.linux-x86_64/egg/elasticsearch/connection/base.py\", line 125, in _raise_error",
      "    raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)",
      "RequestError: TransportError(400, u'search_phase_execution_exception', u'No mapping found for [until] in order to sort on')"
    ],
    "data": {
      "rule": "Many Logs"
    },
    "@timestamp": "2017-11-16T17:40:09.347318Z"
  },
  "fields": {
    "@timestamp": [
      "2017-11-16T17:40:09.347Z"
    ]
  },
  "sort": [
    1510854009347
  ]
}

@emhlbmc
Copy link
Author

emhlbmc commented Nov 17, 2017

@CodingSpiderFox in elasticsearch version 6.0 you can only have one type in mapping of new created index, if you have more than one doc_type, you will get the error 'Rejecting mapping update to [your_index] as the final mapping would have more than 1 type';

Unfortunately the project contains condition code and create 4 or more types to separate the data, such as elastalert_error ,elastalert_status ,past_elastalert ...

This pull request only works with the old index you created before 6.0, and you can use the old index in version 6.0, too.

reference: Removal of mapping types

@spotlesscoder
Copy link
Contributor

How can I use the old index? Do I have to start my elastic 5 stack, then create the old index with an old version of elastalert, then upgrade to elastic 6 and then start this new version of elastalert?

@emhlbmc
Copy link
Author

emhlbmc commented Nov 17, 2017

@CodingSpiderFox yes, you are right.

@spotlesscoder
Copy link
Contributor

How can I upgrade from ES5 to ES6? Is there a proposed upgrade procedure by the developers of the ELK stack?

@emhlbmc
Copy link
Author

emhlbmc commented Nov 18, 2017

@CodingSpiderFox It's not the main topic here, please have a look at ELK official website.

@Jensvandecasteele
Copy link

ES6 only allows 1 mapping type per index. My suggestion would be to create one index for each doctype and instead of referring to the same writeback_index, refer to the doctype index. Only requires a few code changes.

Datatype 'string' does not exist anymore, use 'keyword' instead and index: 'not_analyzed' can be removed when you define your string as 'keyword'.

Got it to work on this pull request with the mentioned changes.

@chandru9279
Copy link

chandru9279 commented Nov 19, 2017

The elastalert.egg-info generated folder needs to be cleared/deleted.

I had to re-run python setup.py install in your virtual env

before the setup worked with ES6

I had to pip install elasticsearch==6.0.0.

@spotlesscoder
Copy link
Contributor

spotlesscoder commented Nov 22, 2017

Is anyone working on native elastic 6 support already?

@Jensvandecasteele are you planning to make a pull request with the changes you made?

@danielpops: In case no one is already working on full elastic 6 support, I could try to do that. But I need to be sure that if I do a pull request with working elastic 6 support, it will be merged in an appropriate timeframe.

When I look at the open pull requests, I get the impression that your organization does not care that much about contributors spending their time to improve elastalert. There are several open PRs which didn't even receive a comment by a maintainer of yelp/elastalert (for example #867 #834). I think this is not a good way to honour the time people spend on doing development here.

To close an issue or a PR without solving/merging it could also be an acceptable action sometimes like for example if an issue is rather a question on how to use elastalert (could be discussed answered in a forum, stackoverflow, etc. if you want the users to do it that way). As long as you state the reason, this would be better than not even answering.

@Jensvandecasteele
Copy link

I started out with a clean ES6 stack. My fixes are a little quick and dirty at some points, but at least it's working. I will have a look this weekend to put the code online.

@spotlesscoder
Copy link
Contributor

@Jensvandecasteele great, thanks!

@Jensvandecasteele
Copy link

Code has been checked in (pull request on Lunatictwo's fork). It's definitely not a final solution, but it gets you going if you just want to try it out on a clean ES6 stack.

@spotlesscoder
Copy link
Contributor

Will try it out tomorrow if I find the time

@spotlesscoder
Copy link
Contributor

It works with elastic stack 6.0.0-beta2 (I used beta2 because I had other problems with Kibana and I thought they weren't there in 6.0.0-beta2 .. now testing 6.0.0 stable)

@spotlesscoder
Copy link
Contributor

spotlesscoder commented Nov 27, 2017

Works with 6.0.0, too but now the ticket seems to be created for every match of the rule:

# Alert when the rate of events exceeds a threshold

# (Required)
# Rule name, must be unique
name: Host Down

type: any

# (Required)
# Index to search, wildcard supported
index: heartbeat-*

query_key: monitor

filter:
  - term:
      monitor.status: down

# (Required)
# The alert is use when a match is found
alert:
- "debug"
- "jira"

jira_label: ["host_down"]
jira_priority: 1

jira_bump_tickets: true
jira_bump_after_inactivity: 0

jira_bump_not_in_statuses:
  - Resolved
  - Closed

jira_issuetype: "Story"

import: "jira_settings.txt"

image

Obviously, the issues only differ in the microseconds field:

ElastAlert: {u'status': u'down', u'name': u'icmp', u'ip': u'<IP>', u'duration': {u'us': 5000109}, u'type': u'icmp', u'id': u'icmp-icmp-ip@<IP>'}
vs.
ElastAlert: {u'status': u'down', u'name': u'icmp', u'ip': u'<IP>', u'duration': {u'us': 5000117}, u'type': u'icmp', u'id': u'icmp-icmp-ip@<IP>'}

a) Is that a known issue?
b) Is it obvious to you that this is because the implementation is not production ready?

@Jensvandecasteele
Copy link

A) Maybe, I only started digging into Elastalert last week. I did however come across the same phenomenon. When I was using a term query I received 1 alert per match, when I used a query string I only received 1. Try using a query string instead. The reason why I changed was because multiple terms caused query issues.

B) It's possible, although I did not change anything about that part as far as I know.

@spotlesscoder
Copy link
Contributor

@Jensvandecasteele yes, maybe that's what my colleague tried to tell me. Something with the queries has changed and needs to be adapted, too, he said.

@Qmando
Copy link
Member

Qmando commented Nov 29, 2017

@CodingSpiderFox

I have been super bad about responding to the pull requests, sorry about that. I've not completely forgotten about them though and I've been meaning to go through all the pending ones and either comment/accept/close. I've introduced a number of bugs by not thoroughly going through them, so I'd like to be careful.

We don't use ES 6, Kibana 5, and aren't actively adding any additional features right now, so it's unfortunately taken a back burner for the time being.

What I need to do (and should have done a long time ago) is create a development branch and ask people to merge into that instead of master, so I can stage changes together more easily without worrying about breaking things for anyone who clones from master instead of pulling a release.

Thank you all for the contributions, I know it's frustrating to have them sit unmerged, I'll take responsibility for that.

@spotlesscoder
Copy link
Contributor

spotlesscoder commented Nov 29, 2017

@Qmando thanks for your effort and undestanding.

I think creating a development branch is a good idea :)
Maybe even two branches, one for Kibana 5/ES5 and one for Kibana 6 /ES6 but the decision is up to you

@spotlesscoder
Copy link
Contributor

spotlesscoder commented Nov 29, 2017

@Jensvandecasteele: The same issue still occurs with with the query_string. My rule looks like this now:

# Alert when the rate of events exceeds a threshold

# (Required)
# Rule name, must be unique
name: Host Down

type: any

# (Required)
# Index to search, wildcard supported
index: heartbeat-*

query_key: monitor

filter:
- query:
    query_string:
      query: "monitor.status: down"

# (Required)
# The alert is use when a match is found
alert:
- "debug"
- "jira"

jira_label: ["host_down"]
jira_priority: 1

jira_label: ["host_down"]
jira_priority: 1

jira_bump_tickets: true
jira_bump_after_inactivity: 0

jira_bump_not_in_statuses:
  - Resolved
  - Closed

import: "jira_settings.txt"

The result still looks like this:

image

Elastalert gives the following error message:

2017-11-29 11:53:40,664 DEBG 'elastalert' stderr output:
ERROR:root:Error while searching for JIRA ticket using jql 'project=<HIDDEN_PROJECT> AND summary~"ElastAlert: {u'status': u'down', u'name': u'icmp', u'ip': u'<HIDDEN_IP>', u'duration': {u'us': 5000129}, u'type': u'icmp', u'id': u'icmp-icmp-ip@<HIDDEN_IP>'} matched Host Down" and created >= "2017-10-30" and status not in (Resolved,Closed)': JiraError HTTP 400 url: http://<HIDDEN_IP_OF_ELK_STACK_SERVER>/rest/api/2/search?jql=project%3D<HIDDEN_PROJECT>+AND+summary~%22ElastAlert%3A+%7Bu%27status%27%3A+u%27down%27%2C+u%27name%27%3A+u%27icmp%27%2C+u%27ip%27%3A+u%27<HIDDEN_IP>%27%2C+u%27duration%27%3A+%7Bu%27us%27%3A+5000129%7D%2C+u%27type%27%3A+u%27icmp%27%2C+u%27id%27%3A+u%27icmp-icmp-ip%40<HIDDEN_IP>%27%7D+matched+Host+Down%22+and+created+%3E%3D+%222017-10-30%22+and+status+not+in+%28Resolved%2CClosed%29&validateQuery=True&startAt=0&maxResults=50
	text: The text query 'ElastAlert: {u'status': u'down', u'name': u'icmp', u'ip': u'<HIDDEN_IP>', u'duration': {u'us': 5000129}, u'type': u'icmp', u'id': u'icmp-icmp-ip@<HIDDEN_IP>'} matched Host Down' for field 'summary' is not valid: probably your range query is incorrect.
	
	response headers = {'X-AUSERNAME': 'elastalert', 'X-AREQUESTID': '713x15600x1', 'X-Content-Type-Options': 'nosniff', 'Content-Encoding': 'gzip', 'Transfer-Encoding': 'chunked', 'X-Seraph-LoginReason': 'OK', 'Vary': 'User-Agent', 'X-ASESSIONID': '1idpo0f', 'Connection': 'close', 'Cache-Control': 'no-cache, no-store, no-transform', 'Date': 'Wed, 29 Nov 2017 11:53:40 GMT', 'Content-Type': 'application/json;charset=UTF-8', 'X-ASEN': 'SEN-L10731268'}
	response text = {"errorMessages":["The text query 'ElastAlert: {u'status': u'down', u'name': u'icmp', u'ip': u'<HIDDEN_IP>', u'duration': {u'us': 5000129}, u'type': u'icmp', u'id': u'icmp-icmp-ip@<HIDDEN_IP>'} matched Host Down' for field 'summary' is not valid: probably your range query is incorrect."],"errors":{}}
Traceback (most recent call last):
  File "elastalert/alerts.py", line 709, in find_existing_ticket
    issues = self.client.search_issues(jql)
  File "/usr/lib/python2.7/site-packages/jira-1.0.10-py2.7.egg/jira/client.py", line 1747, in search_issues
    issues = self._fetch_pages(Issue, 'issues', 'search', startAt, maxResults, search_params)
  File "/usr/lib/python2.7/site-packages/jira-1.0.10-py2.7.egg/jira/client.py", line 397, in _fetch_pages
    resource = self._get_json(request_path, params=page_params, base=base)
  File "/usr/lib/python2.7/site-packages/jira-1.0.10-py2.7.egg/jira/client.py", line 2172, in _get_json
    r = self._session.get(url, params=params)
  File "/usr/lib/python2.7/site-packages/jira-1.0.10-py2.7.egg/jira/resilientsession.py", line 150, in get
    return self.__verb('GET', url, **kwargs)
  File "/usr/lib/python2.7/site-packages/jira-1.0.10-py2.7.egg/jira/resilientsession.py", line 146, in __verb
    raise_on_error(response, verb=verb, **kwargs)
  File "/usr/lib/python2.7/site-packages/jira-1.0.10-py2.7.egg/jira/resilientsession.py", line 56, in raise_on_error
    r.status_code, error, r.url, request=request, response=r, **kwargs)
JIRAError: JiraError HTTP 400 url: http://<HIDDEN_IP_OF_ELK_STACK_SERVER>/rest/api/2/search?jql=project%3D<HIDDEN_PROJECT>+AND+summary~%22ElastAlert%3A+%7Bu%27status%27%3A+u%27down%27%2C+u%27name%27%3A+u%27icmp%27%2C+u%27ip%27%3A+u%27<HIDDEN_IP>%27%2C+u%27duration%27%3A+%7Bu%27us%27%3A+5000129%7D%2C+u%27type%27%3A+u%27icmp%27%2C+u%27id%27%3A+u%27icmp-icmp-ip%40<HIDDEN_IP>%27%7D+matched+Host+Down%22+and+created+%3E%3D+%222017-10-30%22+and+status+not+in+%28Resolved%2CClosed%29&validateQuery=True&startAt=0&maxResults=50
	text: The text query 'ElastAlert: {u'status': u'down', u'name': u'icmp', u'ip': u'<HIDDEN_IP>', u'duration': {u'us': 5000129}, u'type': u'icmp', u'id': u'icmp-icmp-ip@<HIDDEN_IP>'} matched Host Down' for field 'summary' is not valid: probably your range query is incorrect.
	
	response headers = {'X-AUSERNAME': 'elastalert', 'X-AREQUESTID': '713x15600x1', 'X-Content-Type-Options': 'nosniff', 'Content-Encoding': 'gzip', 'Transfer-Encoding': 'chunked', 'X-Seraph-LoginReason': 'OK', 'Vary': 'User-Agent', 'X-ASESSIONID': '1idpo0f', 'Connection': 'close', 'Cache-Control': 'no-cache, no-store, no-transform', 'Date': 'Wed, 29 Nov 2017 11:53:40 GMT', 'Content-Type': 'application/json;charset=UTF-8', 'X-ASEN': 'SEN-L10731268'}
	response text = {"errorMessages":["The text query 'ElastAlert: {u'status': u'down', u'name': u'icmp', u'ip': u'<HIDDEN_IP>', u'duration': {u'us': 5000129}, u'type': u'icmp', u'id': u'icmp-icmp-ip@<HIDDEN_IP>'} matched Host Down' for field 'summary' is not valid: probably your range query is incorrect."],"errors":{}}

2017-11-29 11:53:40,749 DEBG 'elastalert' stderr output:
INFO:elastalert:Opened Jira ticket: <HIDDEN_PROJECT>-35

Could it be that this is a type error in the python source code? Looks like the String is not handeled correctly because the {u'status': u'down', u'name': u'icmp', u'ip': u'<HIDDEN_IP>', u'duration': {u'us': 5000129}, u'type': u'icmp', u'id': u'icmp-icmp-ip@<HIDDEN_IP>'}. All the brackets and Unicode identifiers u'don't belong there

@xaka
Copy link

xaka commented Dec 11, 2017

@Qmando what about having a quick workaround in terms of enforcing "is_five" through configuration file? In our case we don't do much of filtering, but rely on documents count, so just replacing "filtered" with "bool" should be sufficient to pass API validation until proper support for ES 6.x is implemented.

@ahtoluuri
Copy link

ahtoluuri commented Dec 13, 2017

also, it is worth pointing out that in my case the indexes were not created automatically, so I had to go into
<python_dir>/site-packages/elastalert-0.1.21-py2.7.egg/elastalert
and run python create_index.py myself
and then got it working on clean 6.0.0

@minminmsn
Copy link

It's very good. It solves my problem. Except for this type of rule:doc_type must be specified.

@tebeco
Copy link

tebeco commented Dec 20, 2017

Is there somewhere a list of the pending blocking "stuff" to follow for the release ?

@spotlesscoder
Copy link
Contributor

spotlesscoder commented Jan 5, 2018

@Qmando, @snuids : I saw there is a branch "es6_snuids (#1472)" . Any plans when this can be merged into a release version?

@Qmando
Copy link
Member

Qmando commented Jan 5, 2018

Thank you @Lunatictwo and others for your work here, it's greatly appreciated, I've merged in #1472 which was based off of these changes.

@phoenixjun
Copy link

I am having issue when running the elastalert in docker container against ES6.1.2 any work around for this

@bitsensor/elastalert@0.0.9 start /opt/elastalert-server
sh ./scripts/start.sh "/opt/elastalert/logs/logs.txt"

05:42:29.159Z INFO elastalert-server: Config: No config.dev.json file was found in /opt/elastalert-server/config/config.dev.json.
05:42:29.162Z INFO elastalert-server: Config: Proceeding to look for normal config file.
05:42:29.162Z INFO elastalert-server: Config: A config file was found in /opt/elastalert-server/config/config.json. Using that config.
05:42:29.175Z INFO elastalert-server: Router: Listening for GET request on /.
05:42:29.175Z INFO elastalert-server: Router: Listening for GET request on /status.
05:42:29.176Z INFO elastalert-server: Router: Listening for GET request on /status/control/:action.
05:42:29.176Z INFO elastalert-server: Router: Listening for GET request on /status/errors.
05:42:29.176Z INFO elastalert-server: Router: Listening for GET request on /rules.
05:42:29.179Z INFO elastalert-server: Router: Listening for GET request on /rules/:id.
05:42:29.179Z INFO elastalert-server: Router: Listening for POST request on /rules/:id.
05:42:29.179Z INFO elastalert-server: Router: Listening for DELETE request on /rules/:id.
05:42:29.179Z INFO elastalert-server: Router: Listening for GET request on /templates.
05:42:29.180Z INFO elastalert-server: Router: Listening for GET request on /templates/:id.
05:42:29.180Z INFO elastalert-server: Router: Listening for POST request on /templates/:id.
05:42:29.180Z INFO elastalert-server: Router: Listening for DELETE request on /templates/:id.
05:42:29.180Z INFO elastalert-server: Router: Listening for POST request on /test.
05:42:29.180Z INFO elastalert-server: Router: Listening for GET request on /config.
05:42:29.181Z INFO elastalert-server: Router: Listening for POST request on /config.
05:42:29.187Z INFO elastalert-server: ProcessController: Starting ElastAlert
05:42:29.187Z INFO elastalert-server: ProcessController: Creating index
05:42:32.950Z ERROR elastalert-server:
ProcessController: Traceback (most recent call last):
File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
"main", fname, loader, pkg_name)
File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
exec code in run_globals
File "/opt/elastalert/elastalert/create_index.py", line 169, in
main()
File "/opt/elastalert/elastalert/create_index.py", line 153, in main
es.indices.put_mapping(index=index, doc_type='elastalert', body=es_mapping)
File "build/bdist.linux-x86_64/egg/elasticsearch/client/utils.py", line 73, in _wrapped
File "build/bdist.linux-x86_64/egg/elasticsearch/client/indices.py", line 282, in put_mapping
File "build/bdist.linux-x86_64/egg/elasticsearch/transport.py", line 312, in perform_request
File "build/bdist.linux-x86_64/egg/elasticsearch/connection/http_requests.py", line 90, in perform_request
File "build/bdist.linux-x86_64/egg/elasticsearch/connection/base.py", line 125, in _raise_error
elasticsearch.exceptions.RequestError: TransportError(400, u'mapper_parsing_exception', u'No handler for type [string] declared on field [aggregate_id]')

@snuids
Copy link
Contributor

snuids commented Mar 15, 2018

I am using it against 6.1 and 6.2 without any problem. I wrote a small DIY in my blog on how I dockerized ElastAlert here: https://mannekentech.com/2018/01/19/using-elastalert/

Cheers,

Arnaud

@snuids
Copy link
Contributor

snuids commented Mar 15, 2018 via email

@Qmando
Copy link
Member

Qmando commented Mar 15, 2018

@phoenixjun
There could have been an error while your created the index. aggregate_id should be mapped. Try deleting the elastalert_* indexes and rerunning elastalert-create-index.

@cherubic
Copy link

@snuids I see you blog and follow it. my elasticsearch version is 6.2.4, but when I run create_index.py and enter string but it error. below is my enter and output, do you know why?

Thanks.

Enter Elasticsearch host: localhost
Enter Elasticsearch port: 9200
Use SSL? t/f: f
Enter optional basic-auth username (or leave blank): elastic
Enter optional basic-auth password (or leave blank):
Enter optional Elasticsearch URL prefix (prepends a string to the URL of every request): http://localhost:9200
New index name? (Default elastalert_status)
Name of existing index to copy? (Default None)
Traceback (most recent call last):
File "create_index.py", line 272, in
main()
File "create_index.py", line 128, in main
esversion = es.info()["version"]["number"]
File "/Library/Python/2.7/site-packages/elasticsearch/client/utils.py", line 76, in _wrapped
return func(*args, params=params, **kwargs)
File "/Library/Python/2.7/site-packages/elasticsearch/client/init.py", line 241, in info
return self.transport.perform_request('GET', '/', params=params)
File "/Library/Python/2.7/site-packages/elasticsearch/transport.py", line 314, in perform_request
status, headers_response, data = connection.perform_request(method, url, params, body, headers=headers, ignore=ignore, timeout=timeout)
File "/Library/Python/2.7/site-packages/elasticsearch/connection/http_requests.py", line 90, in perform_request
self._raise_error(response.status_code, raw_data)
File "/Library/Python/2.7/site-packages/elasticsearch/connection/base.py", line 125, in _raise_error
raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info)
elasticsearch.exceptions.NotFoundError: TransportError(404, u'index_not_found_exception', u'no such index')

@snuids
Copy link
Contributor

snuids commented May 10, 2018

@cherubic

It looks like a problem in your elastic search installation. The command es.info() performs an http get on the address http://localhost:9200.

If you load this address in your browser, you should receive a json as answer. If you cannot access this address via your browser, it cannot work via the create_index command.

Cheers,

@cherubic
Copy link

@snuids

thank you for your reply.
I can receive a json as answer when I visit http://localhost:9200 in browser. I think my elasticsearch server is work well, I already test it and connect it with kibana. because of this I think that is wrong with the enter string? and where to check it correct?

Thanks.

@Qmando
Copy link
Member

Qmando commented May 11, 2018

@cherubic Don't enter anything for Enter optional Elasticsearch URL prefix. I'll change this to be more clear. This is NOT the url, but a prefix that it would try to add to all URLs. Just hit enter and skip that.

I'll edit this to be more clear. I think it's pretty uncommon that people use this.

@Amorik
Copy link

Amorik commented May 16, 2018

Note that I am also having the same issues as @cherubic and am not entering anything in the prefix. A small self written script also confirms the the elasticsearch python modules works for es.info().

from elasticsearch import Elasticsearch
es = Elasticsearch(["http://localhost:9200"])
print(es.info())
{u'cluster_name': u'MYCLUSTER', u'cluster_uuid': u'9oZn0h7SQEqH9OY9oZv5Ag', u'version': {u'build_date': u'2018-04-12T20:37:28.497551Z', u'minimum_wire_compatibility_version': u'5.6.0', u'build_hash': u'ccec39f', u'number': u'6.2.4', u'minimum_index_compatibility_version': u'5.0.0', u'build_snapshot': False, u'lucene_version': u'7.2.1'}, u'name': u'MYHOSTNAME', u'tagline': u'You Know, for Search'}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet