old gc can not recover memory #27922
Description
Elasticsearch version (bin/elasticsearch --version):
5.3.0
Plugins installed: []
None
JVM version (java -version):
1.8.0_66
OS version (uname -a if on a Unix-like system):
CentOS7
Linux 9ba9ad486a9a 3.10.0-514.21.2.el7.x86_64 #1 SMP Tue Jun 20 12:24:47 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
Description of the problem including expected versus actual behavior:
Steps to reproduce:
We are running a one node elaticsearch cluster with a heap size of 20GB. Sometimes the elasticsearch node is suffering from old gc and can not recover. We dump the heap and see something we can not understand.
-
Can not find classes that use a lot of memory
segment memory, query cache, doc value, fielddata is the GET /_nodes/stats api expose.
-
segment memory
org.apache.lucene.index.SegmentReader uses the most memory, about 4.7GB -
query cache
org.elasticsearch.indices.IndicesQueryCache$ElasticsearchLRUQueryCache uses about 1GB -
doc value
org.apache.lucene.codecs.lucene54DocValueProducer uses about 2GB -
fielddata
org.elasticsearch.index.fielddata.ordinals.InternalGlobalOrdinalsIndexFieldData uses about 700MB
So all these fields uses 8.5G heap size, How is the other heap size used? Why old gc can not fully recover memory?
Provide logs (if relevant):
Activity
s1monw commentedon Dec 20, 2017
can you give us more information about the system in general. ie do you have massive mappings, large number of indices. massive synonym files etc. Can you try to dig where these massive hashmaps are hold?
junkainiu commentedon Dec 21, 2017
@s1monw We do have about 10000 indices in the elasticsearch node, and have a mapping containing about 60 feilds, most of which are keyword type or long type. Not sure about the hashmaps.
jpountz commentedon Dec 21, 2017
60 fields is fine, but 10k indices is too much. This correlates with your heap dump which says there are about 5M segments (SegmentCoreReaders). The reason is that each segment needs to maintain its own metadata, so you should rather have fewer larger segments, and the easiest way to do this is to reduce your number of shards per node, which in turn is probably most easily done by reducing the number of indices. Maybe some of these indices could be merged together? If you use time-based indices, maybe some daily indices could be made monthly indices,etc.