I'm assigning this to myself because I recognize the elements in the stacktrace but I'm not going to have a look until the at least tomorrow morning.

@moshe could you post a gist of the entire stack overflow? It is usually useful to have the root.

Hi @nik9000, thanks for your quick response.
I didn't got stacktrace in the log file, just the error and then the lots of isFinite method call errors.

So reproduced....

@moshe this stack overflow can happen when searching a very big regular expression or when using a syntax for regexp that can lead to an explosion of states. Are you using query_string or regexp queries ? Are you searching regular expressions explicitely ?
For instance the following query:

POST /test/_search
{
  "query": {
    "regexp": {
      "test": "t{1,9500}"
    }
  }
}

fails with a stack overflow error. We have a protection against the explosion of number of states in a non deterministic automaton but for a deterministic one we don't check the size.
This is a known issue in Lucene:
https://issues.apache.org/jira/browse/LUCENE-5659
... but the fix is controversial. Let's check first why you're hitting this.

One of our production cluster experienced the same issue recently due to the abuse of regex/fuzzy queries by developers, but could it be better for Elasticsearch (or maybe Lucene) to set a limit like that on non deterministic automaton, such that the cluster won't be entirely brought down?

Thanks!

After more in-depth investigation, our particular issue was found being caused by prefix query on a very long query string. By looking at source code, I see PrefixQuery does not put a limit on maxDeterminizedStates when instantiate an AutomatonQuery.

public class PrefixQuery extends AutomatonQuery {

  /** Constructs a query for terms starting with <code>prefix</code>. */
  public PrefixQuery(Term prefix) {
    // It's OK to pass unlimited maxDeterminizedStates: the automaton is born small and determinized:
    super(prefix, toAutomaton(prefix.bytes()), Integer.MAX_VALUE, true);
    if (prefix == null) {
      throw new NullPointerException("prefix must not be null");
    }
  }

Is it sensible to limit prefix length & maxDeterminizedStates here?

Thanks for the investigation @xgwu . The automaton is already determinized so the maxDeterminizedStates would not catch very long string here. I think the fix is to limit the input on the ES side for queries that uses an automaton. This would not catch regex query like t{1,9500} but I have the feeling that most of the problems described in this issue are just about very long strings not filtered.

I still think the fix is to put protections in Lucene to prevent the stackoverflow or to rewrite the method so it isn't recursive. I don't think we can allow any queries to shoot the node like this. We could catch the StackOverflow and try to recover but I believe those errors are like OOM, hard to be sure that you've fully recovered from it. If we investigate that to the point where we are sure we have recovered from it then we can just catch it and Lucene doesn't have to change.

We filter long strings and do not use regex queries like mentioned above. We have no idea which query produces this error but it kills elastic several times a day. Any clue what kind of query shoots the node? Would be great if this bug could be fixed as soon as possible.

@hbrxa it sounds like you are suffering from a different bug. please open a new issue with all of the relevant details.

+1

We had a similar issue (on v5.3.0) , coming from kibana users running a simple regex. The regex doesn't even have to really run as search in order to cause the StackOverflowException - just running _validate on such a query cause the same issue.

To reproduce it just type in kibana's Discover:

/.{10000,}/

This issue has been fixed in Lucene:
https://issues.apache.org/jira/browse/LUCENE-7914
... and will be available in es 6.0
The workaround for es 5.x is to check the size of the prefix on the client side in order to prevent the stack overflow (something like a few hundreds chars should be a good limit).