New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Strange IllegalReferenceCountException #4435
Comments
@doom369 can you show me all the handlers you have in your pipeline ? |
@doom369 I am pretty sure you broke the refcnt somewhere in your handlers, somewhere near composite buffers... |
@ninja- how could that be if in pipeline where error happen there is no my code that manages refcnt at any way. Netty code itself manages this. Also server was running for a weeks with load ~700 req/sec and this is only such error I found. |
One more, but look like in another pipeline (with SSL Handler)
|
@doom369 - It seems that buffer which is owned by Is there any place where you don't copy the contents of the buffer and instead use the buffer (or a slice of the buffer) directly? For example writing a buffer you decode to a channel. Can you reproduce the issue? If so it would be good to know what the reference count I don't see any SSL initialization in |
No. I deal with netty buffers only in MessageDecoder and MessageEncoder. In all other handlers I work with MessageBase that those classes produce.
No. It is just appearing in production system.
I have 4 different pipelines. 3 of them using SSL handlers. 1 of them (HardwareServer) doesn't. First stack I posted in ticket is without SSLHandler. So that means that exception came from pipeline of HardwareServer. All pipelines using MessageEncoder and MessageDecoder. |
very strange... Currently I have no idea what's wrong :/ |
It is interesting that the original leak report you posted stems from the condition if your application code consumes all data in the if (cumulation != null && !cumulation.isReadable()) {
numReads = 0;
cumulation.release();
cumulation = null;
} |
@Scottmitch I'll do new deployment (that is planned) of 4.0.33 version. And will monitor it. If no error I'll close issue, otherwise will add more debug. |
4.0.33 didn't help. But I was able to reproduce issue on QA server. Will try to prepare some simple setup for you. |
@doom369 thanks! |
@normanmaurer sorry for huge delay. So finally I had a time to play with that.
So regarding reproducing. Here is a server jar. To run please use :
Also please checkout this project. Find test SimplePerformanceTest.java remove @ignore annotation from method Now in directory from where you where running jar file you'll find I have this problem on 3 envs. All Ubuntu, 12.04, 14.04. Java version "1.8.0_40", "1.8.0_45" Please let me know if you need more info. |
@doom369 thanks will check! |
@doom369 can you provide me with a version as well where I could include a different netty version to test ? Like not an "all in one jar". |
@normanmaurer sure, server.jar I provided is build from those project with
And you may run
All netty versions are in main pom file. You can find output in |
@doom369 this only happens with epoll ? |
epoll + open ssl. epoll + JDK SSL - not reproducible. |
ok gotcha! |
@doom369 ok can reproduce with your stuff... will come back to you :) |
nice =) |
@doom369 so good thing is I know what happens... the bad, I just not understand why 👎 |
=) Let me know if I can help somehow. |
@doom369 I now know what happens and also why (yay). Working on a fix... to make it short it has to do with the register/deregister that you do in your handler. |
@normanmaurer Hm... I had a feeling regarding that. Should tell you before. Glad you found problem. |
@normanmaurer I was not able to build epoll module, maybe you have ready build somewhere? |
Nope :( what is the error?
|
@normanmaurer It was issue with JDK. Fixed, made new build, tested with it, seems not reproducible any more. Thanks! |
Motivation: As a user may call deregister() from within any method while doing processing in the ChannelPipeline, we need to ensure we do the actual deregister operation later. This is needed as for example, we may be in the ByteToMessageDecoder.callDecode(...) method and so still try to do processing in the old EventLoop while the user already registered the Channel to a new EventLoop. Without delay, the deregister operation this could lead to have a handler invoked by different EventLoop and so threads. Modifications: Ensure the actual deregister will be done later on and not directly when invoked. Result: Calling deregister() within ByteToMessageDecoder.decode(..) is safe.
Motivation: As a user may call deregister() from within any method while doing processing in the ChannelPipeline, we need to ensure we do the actual deregister operation later. This is needed as for example, we may be in the ByteToMessageDecoder.callDecode(...) method and so still try to do processing in the old EventLoop while the user already registered the Channel to a new EventLoop. Without delay, the deregister operation this could lead to have a handler invoked by different EventLoop and so threads. Modifications: Ensure the actual deregister will be done later on and not directly when invoked. Result: Calling deregister() within ByteToMessageDecoder.decode(..) is safe.
Motivation: As a user may call deregister() from within any method while doing processing in the ChannelPipeline, we need to ensure we do the actual deregister operation later. This is needed as for example, we may be in the ByteToMessageDecoder.callDecode(...) method and so still try to do processing in the old EventLoop while the user already registered the Channel to a new EventLoop. Without delay, the deregister operation this could lead to have a handler invoked by different EventLoop and so threads. Modifications: Ensure the actual deregister will be done later on and not directly when invoked. Result: Calling deregister() within ByteToMessageDecoder.decode(..) is safe.
…entLoop. Motivation: As a user may call deregister() from within any method while doing processing in the ChannelPipeline, we need to ensure we do the actual deregister operation later. This is needed as for example, we may be in the ByteToMessageDecoder.callDecode(...) method and so still try to do processing in the old EventLoop while the user already registered the Channel to a new EventLoop. Without delay, the deregister operation this could lead to have a handler invoked by different EventLoop and so threads. Modifications: Ensure the actual deregister will be done later on and not directly when invoked. Result: Calling deregister() within ByteToMessageDecoder.decode(..) is safe.
Hello. 4.0.32.Final, Ubuntu 14.04. I have a production system running for few weeks. Was reviewing logs and found this:
Not sure is that netty bug or my own. Decoder code - https://github.com/blynkkk/blynk-server/blob/master/common/src/main/java/cc/blynk/common/handlers/common/decoders/MessageDecoder.java
The text was updated successfully, but these errors were encountered: