Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

重启tomcat的时候DubboShutdownHook会报一个非常讨厌的WARN #333

Closed
jiakai0419 opened this issue Oct 18, 2016 · 13 comments
Closed

重启tomcat的时候DubboShutdownHook会报一个非常讨厌的WARN #333

jiakai0419 opened this issue Oct 18, 2016 · 13 comments
Milestone
2.5.4.M1

Comments

@jiakai0419
Copy link

jiakai0419 commented Oct 18, 2016 •
edited by oldratlee

现象

每当我重启tomcat的时候，就会报下面的warning。

2016-10-18 14:30:09.470  INFO DubboShutdownHook ZookeeperRegistry:485 -  [DUBBO] Destroy registry:zookeeper://172.21.134.7:2005/com.alibaba.dubbo.registry.RegistryService?application=stormsun&dubbo=2.8.4&interface=com.alibaba.dubbo.registry.RegistryService&owner=kaijia&pid=868&timestamp=1476760120506, dubbo version: 2.8.4, current host: 127.0.0.1
2016-10-18 14:30:09.471  WARN DubboShutdownHook ZookeeperRegistry:440 -  [DUBBO] java/util/concurrent/ScheduledFuture, dubbo version: 2.8.4, current host: 127.0.0.1
java.lang.NoClassDefFoundError: java/util/concurrent/ScheduledFuture
    at com.alibaba.dubbo.registry.support.FailbackRegistry.destroy(FailbackRegistry.java:438)
    at com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry.destroy(ZookeeperRegistry.java:88)
    at com.alibaba.dubbo.registry.support.AbstractRegistryFactory.destroyAll(AbstractRegistryFactory.java:70)
    at com.alibaba.dubbo.config.ProtocolConfig.destroyAll(ProtocolConfig.java:463)
    at com.alibaba.dubbo.config.AbstractConfig$1.run(AbstractConfig.java:452)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: Illegal access: this web application instance has been stopped already. Could not load [java.util.concurrent.ScheduledFuture]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1315)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1178)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1139)
    ... 6 more
Caused by: java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.util.concurrent.ScheduledFuture]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1325)
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1313)
    ... 8 more
2016-10-18 14:30:09.471  INFO ZkClient-EventThread-29-172.21.134.7:2005 ZkEventThread:82 - Terminate ZkClient event thread.
2016-10-18 14:30:09.476  INFO localhost-startStop-1-EventThread ClientCnxn:516 - EventThread shut down
2016-10-18 14:30:09.476  INFO DubboShutdownHook ZooKeeper:544 - Session: 0x157bc04550c0142 closed
2016-10-18 14:30:09.477  INFO DubboShutdownHook DubboProtocol:439 -  [DUBBO] Close dubbo server: /172.21.134.7:20883, dubbo version: 2.8.4, current host: 127.0.0.1
2016-10-18 14:30:09.479  WARN DubboShutdownHook HeaderExchangeServer:255 -  [DUBBO] java/util/concurrent/ScheduledFuture, dubbo version: 2.8.4, current host: 127.0.0.1
java.lang.NoClassDefFoundError: java/util/concurrent/ScheduledFuture
    at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeServer.stopHeartbeatTimer(HeaderExchangeServer.java:251)
    at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeServer.doClose(HeaderExchangeServer.java:145)
    at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeServer.close(HeaderExchangeServer.java:120)
    at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol.destroy(DubboProtocol.java:441)
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper.destroy(ProtocolFilterWrapper.java:66)
    at com.alibaba.dubbo.rpc.protocol.ProtocolListenerWrapper.destroy(ProtocolListenerWrapper.java:72)
    at com.alibaba.dubbo.config.ProtocolConfig.destroyAll(ProtocolConfig.java:469)
    at com.alibaba.dubbo.config.AbstractConfig$1.run(AbstractConfig.java:452)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: Illegal access: this web application instance has been stopped already. Could not load [java.util.concurrent.ScheduledFuture]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1315)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1178)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1139)
    ... 9 more
Caused by: java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.util.concurrent.ScheduledFuture]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1325)
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1313)
    ... 11 more
2016-10-18 14:30:09.487  INFO DubboShutdownHook AbstractServer:147 -  [DUBBO] Close NettyServer bind /0.0.0.0:20883, export /172.21.134.7:20883, dubbo version: 2.8.4, current host: 127.0.0.1
2016-10-18 14:30:09.495  WARN DubboShutdownHook NettyServer:127 -  [DUBBO] org/jboss/netty/util/internal/ExecutorUtil, dubbo version: 2.8.4, current host: 127.0.0.1
java.lang.NoClassDefFoundError: org/jboss/netty/util/internal/ExecutorUtil
    at org.jboss.netty.channel.socket.nio.AbstractNioBossPool.releaseExternalResources(AbstractNioBossPool.java:91)
    at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.releasePools(NioServerSocketChannelFactory.java:223)
    at org.jboss.netty.channel.socket.nio.NioServerSocketChannelFactory.releaseExternalResources(NioServerSocketChannelFactory.java:218)
    at org.jboss.netty.bootstrap.Bootstrap.releaseExternalResources(Bootstrap.java:318)
    at com.alibaba.dubbo.remoting.transport.netty.NettyServer.doClose(NettyServer.java:124)
    at com.alibaba.dubbo.remoting.transport.AbstractServer.close(AbstractServer.java:156)
    at com.alibaba.dubbo.remoting.transport.AbstractServer.close(AbstractServer.java:164)
    at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeServer.close(HeaderExchangeServer.java:121)
    at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol.destroy(DubboProtocol.java:441)
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper.destroy(ProtocolFilterWrapper.java:66)
    at com.alibaba.dubbo.rpc.protocol.ProtocolListenerWrapper.destroy(ProtocolListenerWrapper.java:72)
    at com.alibaba.dubbo.config.ProtocolConfig.destroyAll(ProtocolConfig.java:469)
    at com.alibaba.dubbo.config.AbstractConfig$1.run(AbstractConfig.java:452)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.ClassNotFoundException: Illegal access: this web application instance has been stopped already. Could not load [org.jboss.netty.util.internal.ExecutorUtil]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1315)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1178)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1139)
    ... 14 more
Caused by: java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.jboss.netty.util.internal.ExecutorUtil]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1325)
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1313)
    ... 16 more
2016-10-18 14:30:13.291  INFO localhost-startStop-1 ContextLoader:307 - Root WebApplicationContext: initialization started

猜测和目标

看起来像dubbo执行shutdown hook的时候，web application已经停掉了。
我这个强迫症看这个warning特别难受，在tomcat重启的时候我怎样才能优雅的关闭dubbo，消除掉这个warning。

环境

编译方式

            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.5.1</version>
                <configuration>
                    <source>1.8</source>
                    <target>1.8</target>
                    <encoding>UTF-8</encoding>
                </configuration>
            </plugin>

JRE

java version "1.8.0_11"
Java(TM) SE Runtime Environment (build 1.8.0_11-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.11-b03, mixed mode)

Tomcat

Apache Tomcat Version 8.0.33
@wuwen5
Copy link
Contributor

wuwen5 commented Oct 18, 2016 •
edited by oldratlee

主动自行调用ProtocolConfig.destroyAll();，保障在jvm退出之前调用。
这样当DubboShutdownHook再次执行的时候待销毁资源已经clear了，就不会有异常了。

看起来像dubbo执行shutdown hook的时候，web application已经停掉了。
说明dubbo服务没有优雅停机.

@jiakai0419
Copy link
Author

jiakai0419 commented Oct 18, 2016 •
edited

按照上面的方法修改后，报了下面的WARN。

2016-10-18 21:30:54.854  WARN localhost-startStop-2 RegistryProtocol:132 -  [DUBBO] Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797399783&version=1.0.0 to registry 172.21.134.7:2005, cause: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797399783&version=1.0.0 to zookeeper zookeeper://172.21.134.7:2005/com.alibaba.dubbo.registry.RegistryService?application=stormsun&dubbo=2.8.4&interface=com.alibaba.dubbo.registry.RegistryService&owner=kaijia&pid=6586&timestamp=1476797399775, cause: null, dubbo version: 2.8.4, current host: 127.0.0.1
java.lang.IllegalStateException: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797399783&version=1.0.0 to registry 172.21.134.7:2005, cause: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797399783&version=1.0.0 to zookeeper zookeeper://172.21.134.7:2005/com.alibaba.dubbo.registry.RegistryService?application=stormsun&dubbo=2.8.4&interface=com.alibaba.dubbo.registry.RegistryService&owner=kaijia&pid=6586&timestamp=1476797399775, cause: null
    at com.alibaba.dubbo.registry.support.FailbackRegistry.unregister(FailbackRegistry.java:173)
    at com.alibaba.dubbo.registry.integration.RegistryProtocol$1.unexport(RegistryProtocol.java:130)
    at com.alibaba.dubbo.config.ServiceConfig.unexport(ServiceConfig.java:271)
    at com.alibaba.dubbo.config.spring.AnnotationBean.destroy(AnnotationBean.java:120)
    at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:261)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:972)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:979)
    at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1000)
    at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:976)
    at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:928)
    at org.springframework.web.context.ContextLoader.closeWebApplicationContext(ContextLoader.java:583)
    at org.springframework.web.context.ContextLoaderListener.contextDestroyed(ContextLoaderListener.java:116)
    at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4858)
    at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5474)
    at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:224)
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1424)
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1413)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.alibaba.dubbo.rpc.RpcException: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797399783&version=1.0.0 to zookeeper zookeeper://172.21.134.7:2005/com.alibaba.dubbo.registry.RegistryService?application=stormsun&dubbo=2.8.4&interface=com.alibaba.dubbo.registry.RegistryService&owner=kaijia&pid=6586&timestamp=1476797399775, cause: null
    at com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry.doUnregister(ZookeeperRegistry.java:108)
    at com.alibaba.dubbo.registry.support.FailbackRegistry.unregister(FailbackRegistry.java:160)
    ... 23 more
Caused by: java.lang.NullPointerException
    at org.I0Itec.zkclient.ZkClient$8.call(ZkClient.java:720)
    at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
    at org.I0Itec.zkclient.ZkClient.delete(ZkClient.java:716)
    at com.alibaba.dubbo.remoting.zookeeper.zkclient.ZkclientZookeeperClient.delete(ZkclientZookeeperClient.java:61)
    at com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry.doUnregister(ZookeeperRegistry.java:106)
    ... 24 more
2016-10-18 21:30:54.855  INFO localhost-startStop-2 ZookeeperRegistry:343 -  [DUBBO] Unsubscribe: provider://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&category=configurators&check=false&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797399783&version=1.0.0, dubbo version: 2.8.4, current host: 127.0.0.1
2016-10-18 21:30:54.856  INFO localhost-startStop-2 ZookeeperRegistry:312 -  [DUBBO] Unregister: dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.open.OpenApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.open.OpenApi&methods=queryFriendInfo,queryChatroomInfo,alterChatroomNotice,alterSelfDisplayName,batchSend,alterChatroomName,alterEntityPriority,queryChatrommMemberCount,liberate,applyXRobot,sendImage,sendText,kickSomeBody,manage,fire,sendVoice,transferOwner,send,queryUsernameChatroomInfo&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797400203&version=1.0.0, dubbo version: 2.8.4, current host: 127.0.0.1
2016-10-18 21:30:54.856  WARN localhost-startStop-2 RegistryProtocol:132 -  [DUBBO] Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.open.OpenApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.open.OpenApi&methods=queryFriendInfo,queryChatroomInfo,alterChatroomNotice,alterSelfDisplayName,batchSend,alterChatroomName,alterEntityPriority,queryChatrommMemberCount,liberate,applyXRobot,sendImage,sendText,kickSomeBody,manage,fire,sendVoice,transferOwner,send,queryUsernameChatroomInfo&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797400203&version=1.0.0 to registry 172.21.134.7:2005, cause: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.open.OpenApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.open.OpenApi&methods=queryFriendInfo,queryChatroomInfo,alterChatroomNotice,alterSelfDisplayName,batchSend,alterChatroomName,alterEntityPriority,queryChatrommMemberCount,liberate,applyXRobot,sendImage,sendText,kickSomeBody,manage,fire,sendVoice,transferOwner,send,queryUsernameChatroomInfo&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797400203&version=1.0.0 to zookeeper zookeeper://172.21.134.7:2005/com.alibaba.dubbo.registry.RegistryService?application=stormsun&dubbo=2.8.4&interface=com.alibaba.dubbo.registry.RegistryService&owner=kaijia&pid=6586&timestamp=1476797399775, cause: null, dubbo version: 2.8.4, current host: 127.0.0.1
java.lang.IllegalStateException: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.open.OpenApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.open.OpenApi&methods=queryFriendInfo,queryChatroomInfo,alterChatroomNotice,alterSelfDisplayName,batchSend,alterChatroomName,alterEntityPriority,queryChatrommMemberCount,liberate,applyXRobot,sendImage,sendText,kickSomeBody,manage,fire,sendVoice,transferOwner,send,queryUsernameChatroomInfo&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797400203&version=1.0.0 to registry 172.21.134.7:2005, cause: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.open.OpenApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.open.OpenApi&methods=queryFriendInfo,queryChatroomInfo,alterChatroomNotice,alterSelfDisplayName,batchSend,alterChatroomName,alterEntityPriority,queryChatrommMemberCount,liberate,applyXRobot,sendImage,sendText,kickSomeBody,manage,fire,sendVoice,transferOwner,send,queryUsernameChatroomInfo&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797400203&version=1.0.0 to zookeeper zookeeper://172.21.134.7:2005/com.alibaba.dubbo.registry.RegistryService?application=stormsun&dubbo=2.8.4&interface=com.alibaba.dubbo.registry.RegistryService&owner=kaijia&pid=6586&timestamp=1476797399775, cause: null
    at com.alibaba.dubbo.registry.support.FailbackRegistry.unregister(FailbackRegistry.java:173)
    at com.alibaba.dubbo.registry.integration.RegistryProtocol$1.unexport(RegistryProtocol.java:130)
    at com.alibaba.dubbo.config.ServiceConfig.unexport(ServiceConfig.java:271)
    at com.alibaba.dubbo.config.spring.AnnotationBean.destroy(AnnotationBean.java:120)
    at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:261)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:972)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:979)
    at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1000)
    at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:976)
    at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:928)
    at org.springframework.web.context.ContextLoader.closeWebApplicationContext(ContextLoader.java:583)
    at org.springframework.web.context.ContextLoaderListener.contextDestroyed(ContextLoaderListener.java:116)
    at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4858)
    at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5474)
    at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:224)
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1424)
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1413)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.alibaba.dubbo.rpc.RpcException: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.open.OpenApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.open.OpenApi&methods=queryFriendInfo,queryChatroomInfo,alterChatroomNotice,alterSelfDisplayName,batchSend,alterChatroomName,alterEntityPriority,queryChatrommMemberCount,liberate,applyXRobot,sendImage,sendText,kickSomeBody,manage,fire,sendVoice,transferOwner,send,queryUsernameChatroomInfo&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797400203&version=1.0.0 to zookeeper zookeeper://172.21.134.7:2005/com.alibaba.dubbo.registry.RegistryService?application=stormsun&dubbo=2.8.4&interface=com.alibaba.dubbo.registry.RegistryService&owner=kaijia&pid=6586&timestamp=1476797399775, cause: null
    at com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry.doUnregister(ZookeeperRegistry.java:108)
    at com.alibaba.dubbo.registry.support.FailbackRegistry.unregister(FailbackRegistry.java:160)
    ... 23 more
Caused by: java.lang.NullPointerException
    at org.I0Itec.zkclient.ZkClient$8.call(ZkClient.java:720)
    at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
    at org.I0Itec.zkclient.ZkClient.delete(ZkClient.java:716)
    at com.alibaba.dubbo.remoting.zookeeper.zkclient.ZkclientZookeeperClient.delete(ZkclientZookeeperClient.java:61)
    at com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry.doUnregister(ZookeeperRegistry.java:106)
    ... 24 more
2016-10-18 21:30:54.856  INFO localhost-startStop-2 ZookeeperRegistry:343 -  [DUBBO] Unsubscribe: provider://172.21.134.7:20883/com.baijia.storm.sun.api.open.OpenApi?anyhost=true&application=stormsun&category=configurators&check=false&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.open.OpenApi&methods=queryFriendInfo,queryChatroomInfo,alterChatroomNotice,alterSelfDisplayName,batchSend,alterChatroomName,alterEntityPriority,queryChatrommMemberCount,liberate,applyXRobot,sendImage,sendText,kickSomeBody,manage,fire,sendVoice,transferOwner,send,queryUsernameChatroomInfo&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797400203&version=1.0.0, dubbo version: 2.8.4, current host: 127.0.0.1
2016-10-18 21:30:54.890 ERROR localhost-startStop-1-SendThread(172.21.134.7:2005) ClientCnxn:418 - from localhost-startStop-1-SendThread(172.21.134.7:2005)
java.lang.NoClassDefFoundError: org/apache/zookeeper/server/ZooTrace
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1220)
Caused by: java.lang.ClassNotFoundException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1315)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1178)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1139)
    ... 1 more
Caused by: java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1325)
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1313)
    ... 3 more
2016-10-18 21:30:54.938  INFO DubboShutdownHook AbstractConfig:450 -  [DUBBO] Run shutdown hook now., dubbo version: 2.8.4, current host: 127.0.0.1
2016-10-18 21:30:54.938  INFO DubboShutdownHook AbstractRegistryFactory:63 -  [DUBBO] Close all registries [], dubbo version: 2.8.4, current host: 127.0.0.1
2016-10-18 21:30:59.305  INFO localhost-startStop-1 ContextLoader:307 - Root WebApplicationContext: initialization started

#95
这个issue说原因是消费端一直没有在ZooKeeper注册中心注销
说要改zookeeper版本问题, 我试了下没什么用。

           <dependency>
                <groupId>com.github.sgroschupf</groupId>
                <artifactId>zkclient</artifactId>
                <version>0.1</version>
                <exclusions>
                    <exclusion>
                        <groupId>org.apache.zookeeper</groupId>
                        <artifactId>zookeeper</artifactId>
                    </exclusion>
                </exclusions>
            </dependency>
            <dependency>
                <groupId>org.apache.zookeeper</groupId>
                <artifactId>zookeeper</artifactId>
                <version>3.4.5</version>
            </dependency>

@wuwen5
Copy link
Contributor

wuwen5 commented Oct 19, 2016

这个是因为ProtocolConfig.destroyAll()的时候，资源已经释放，zkclient连接已经关闭，你使用的应该是0.1版本的zkclient，close的时候_connection会置为null， 接着spring容器close的时候，dubbo服务会再次执行释放操作，最终会再次调用到zkclient的 delete方法，报NPE异常。

可以升级zkclient版本到0.5以上，最新版本是0.9，我们目前是0.6.

zkclient 修复记录 sgroschupf/zkclient@0630c9c

注：是group为com.101tec

@jiakai0419
Copy link
Author

更新了zkclient的version, null pointer变成了更清晰的解释ZkClient already closed，针对这个warning我是可以接受的，但是下面报了zk的error看起来像没有优雅的结束zk，这个怎么搞定？

            <dependency>
                <groupId>com.101tec</groupId>
                <artifactId>zkclient</artifactId>
                <version>0.8</version>
            </dependency>

2016-10-21 13:17:21.530  WARN localhost-startStop-2 RegistryProtocol:132 -  [DUBBO] Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=2434&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1477026966843&version=1.0.0 to registry 172.21.134.7:2005, cause: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=2434&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1477026966843&version=1.0.0 to zookeeper zookeeper://172.21.134.7:2005/com.alibaba.dubbo.registry.RegistryService?application=stormsun&dubbo=2.8.4&interface=com.alibaba.dubbo.registry.RegistryService&owner=kaijia&pid=2434&timestamp=1477026966835, cause: ZkClient already closed!, dubbo version: 2.8.4, current host: 127.0.0.1
java.lang.IllegalStateException: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=2434&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1477026966843&version=1.0.0 to registry 172.21.134.7:2005, cause: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=2434&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1477026966843&version=1.0.0 to zookeeper zookeeper://172.21.134.7:2005/com.alibaba.dubbo.registry.RegistryService?application=stormsun&dubbo=2.8.4&interface=com.alibaba.dubbo.registry.RegistryService&owner=kaijia&pid=2434&timestamp=1477026966835, cause: ZkClient already closed!
    at com.alibaba.dubbo.registry.support.FailbackRegistry.unregister(FailbackRegistry.java:173)
    at com.alibaba.dubbo.registry.integration.RegistryProtocol$1.unexport(RegistryProtocol.java:130)
    at com.alibaba.dubbo.config.ServiceConfig.unexport(ServiceConfig.java:271)
    at com.alibaba.dubbo.config.spring.AnnotationBean.destroy(AnnotationBean.java:120)
    at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:261)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:972)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:979)
    at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1000)
    at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:976)
    at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:928)
    at org.springframework.web.context.ContextLoader.closeWebApplicationContext(ContextLoader.java:583)
    at org.springframework.web.context.ContextLoaderListener.contextDestroyed(ContextLoaderListener.java:116)
    at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4858)
    at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5474)
    at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:224)
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1424)
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1413)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.alibaba.dubbo.rpc.RpcException: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=2434&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1477026966843&version=1.0.0 to zookeeper zookeeper://172.21.134.7:2005/com.alibaba.dubbo.registry.RegistryService?application=stormsun&dubbo=2.8.4&interface=com.alibaba.dubbo.registry.RegistryService&owner=kaijia&pid=2434&timestamp=1477026966835, cause: ZkClient already closed!
    at com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry.doUnregister(ZookeeperRegistry.java:108)
    at com.alibaba.dubbo.registry.support.FailbackRegistry.unregister(FailbackRegistry.java:160)
    ... 23 more
Caused by: java.lang.IllegalStateException: ZkClient already closed!
    at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:987)
    at org.I0Itec.zkclient.ZkClient.delete(ZkClient.java:1047)
    at org.I0Itec.zkclient.ZkClient.delete(ZkClient.java:1042)
    at com.alibaba.dubbo.remoting.zookeeper.zkclient.ZkclientZookeeperClient.delete(ZkclientZookeeperClient.java:61)
    at com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry.doUnregister(ZookeeperRegistry.java:106)
    ... 24 more
2016-10-21 13:17:21.531  INFO localhost-startStop-2 ZookeeperRegistry:343 -  [DUBBO] Unsubscribe: provider://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&category=configurators&check=false&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=2434&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1477026966843&version=1.0.0, dubbo version: 2.8.4, current host: 127.0.0.1
2016-10-21 13:17:21.618 ERROR localhost-startStop-1-SendThread(172.21.134.7:2005) ClientCnxn:414 - from localhost-startStop-1-SendThread(172.21.134.7:2005)
java.lang.NoClassDefFoundError: org/apache/zookeeper/server/ZooTrace
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1128)
Caused by: java.lang.ClassNotFoundException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1315)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1178)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1139)
    ... 1 more
Caused by: java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1325)
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1313)
    ... 3 more
2016-10-21 13:17:21.679  INFO DubboShutdownHook AbstractConfig:450 -  [DUBBO] Run shutdown hook now., dubbo version: 2.8.4, current host: 127.0.0.1
2016-10-21 13:17:21.679  INFO DubboShutdownHook AbstractRegistryFactory:63 -  [DUBBO] Close all registries [], dubbo version: 2.8.4, current host: 127.0.0.1
2016-10-21 13:17:25.558  INFO localhost-startStop-1 ContextLoader:307 - Root WebApplicationContext: initialization started

@wuwen5
Copy link
Contributor

wuwen5 commented Oct 21, 2016 •
edited

ZooTrace 视乎仅仅只是个打日志的，不影响。
因为zookeeper client端 这段代码从来没有执行过，只在最后释放的时候打了个日志，所以这个类只有在释放的时候才会被加载，而tomcat容器已经停止不能再load class了。

这个异常可以尝试 规避下。
在tomcat停止之前，你在项目代码中加上一段（保障能执行,例如 写在ProtocolConfig.destroyAll()代码的位置）或者也可以在启动的地方加一下.

   ZooTrace.logTraceMessage(LOG, ZooTrace.getTextTraceLevel(),
                                     "Run shutdown now.");

另外 你再检查下是否有zookeeper包版本冲突.

@jiakai0419
Copy link
Author

jiakai0419 commented Oct 21, 2016 •
edited

现象和猜测

报了两个新的异常，猜测是关闭dubbo的时候他想把现有的任务执行完毕，但是依赖的资源已经销毁了。
我有什么办法能控制销毁顺序吗，让这些异常也避免掉？

我目前是通过这种方法调用destoryAll()的。

@Slf4j
@Component
public class GraceShutdownListener {
    @PreDestroy
    public void graceShutdown() {
        log.info("GRACE SHUTDOWN");
        ZooTrace.logTraceMessage(log, ZooTrace.getTextTraceLevel(), "manual ZooTrace");
        ProtocolConfig.destroyAll();
    }
}

2016-10-21 14:44:08.240 ERROR DubboServerHandler-172.21.134.7:20883-thread-17 OpenApiImpl:97 - SYS_ERROR sunChat[{"weChatMsgType":3,"content":"http://file.gsxservice.com/stormEarth/robotFiles/imgMsg/6801998040@chatroom/342311477032243354.png","addressList":["7815031688@chatroom","7757035007@chatroom","7984035562@chatroom","8188035593@chatroom","7815031688@chatroom","7882043131@chatroom","7905043678@chatroom","8061042899@chatroom","7963049048@chatroom","6778991480@chatroom","6610991914@chatroom","7057992665@chatroom"],"addressType":1}]
redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool

2016-10-21 14:44:08.248  WARN DubboServerHandler-172.21.134.7:20883-thread-17 ChannelEventRunnable:84 -  [DUBBO] ChannelEventRunnable handle RECEIVED operation error, channel is NettyChannel [channel=[id: 0xc3355077, /172.21.134.7:34208 :> /172.21.134.7:20883]], message is Request [id=2660, version=2.0.0, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=send, parameterTypes=[class com.baijia.storm.sun.api.common.proto.SunChat], arguments=[SunChat(weChatMsgType=3, content=http://file.gsxservice.com/stormEarth/robotFiles/imgMsg/6801998040@chatroom/342311477032243354.png, addressList=[7815031688@chatroom, 7757035007@chatroom, 7984035562@chatroom, 8188035593@chatroom, 7815031688@chatroom, 7882043131@chatroom, 7905043678@chatroom, 8061042899@chatroom, 7963049048@chatroom, 6778991480@chatroom, 6610991914@chatroom, 7057992665@chatroom], addressType=1)], attachments={path=com.baijia.storm.sun.api.open.OpenApi, input=697, dubbo=2.8.4, interface=com.baijia.storm.sun.api.open.OpenApi, version=1.0.0}]], dubbo version: 2.8.4, current host: 127.0.0.1
com.alibaba.dubbo.remoting.RemotingException: Failed to send message Response [id=2660, version=2.0.0, status=20, event=false, error=null, result=RpcResult [result=SunApiResponse(code=-3, msg=sys error, data=null), exception=null]] to /172.21.134.7:34208, cause: null
    at com.alibaba.dubbo.remoting.transport.netty.NettyChannel.send(NettyChannel.java:108)
    at com.alibaba.dubbo.remoting.transport.AbstractPeer.send(AbstractPeer.java:51)
    at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:171)
    at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
    at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedChannelException
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:434)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:129)
    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99)
    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
    at org.jboss.netty.channel.Channels.write(Channels.java:725)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
    at com.alibaba.dubbo.remoting.transport.netty.NettyHandler.writeRequested(NettyHandler.java:99)
    at org.jboss.netty.channel.Channels.write(Channels.java:704)
    at org.jboss.netty.channel.Channels.write(Channels.java:671)
    at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248)
    at com.alibaba.dubbo.remoting.transport.netty.NettyChannel.send(NettyChannel.java:98)
    ... 7 more

@wuwen5
Copy link
Contributor

wuwen5 commented Oct 21, 2016

在你的graceShutdown方法中做下连接检测，保障所有已经进来的请求全部吐出去了。

例如

step 1.
AbstractRegistryFactory.destroyAll()
断开注册中心的连接，这样可以保障没有新的请求进入这台负载。

step 2.
连接检测，可以通过dubbo filter 做一个全局的请求计数器，检测所有请求都出去了。
(这一步你可以先sleep 一段时间测试下。)

step 3.
ProtocolConfig.destroyAll()

@jiakai0419
Copy link
Author

jiakai0419 commented Oct 21, 2016 •
edited by oldratlee

It works!
thanks @wuwen5 .

@wuwen5
Copy link
Contributor

wuwen5 commented Oct 21, 2016

@PreDestroy会在spring 容器destroy处理开始之前执行，你的redis连接池应该是由spring容器管理，spring容器会有序销毁资源。

@AnyinSong
Copy link

最好不要在webapp中使用hook，webapp反部署时无法触发是一方面，更重要是的影响元空间回收

@jiakai0419
Copy link
Author

@AnyinSong 可以展开在说一下嘛。
例如想要在webapp中提供dubbo服务，关于grace shutdown有什么更具体的建议。

@AnyinSong
Copy link

如果要在webapp中提供dubbo服务的话那在这个webapp的反部署逻辑中就要销毁所有dubbo资源，不能等到tomcat进程停止时做，比如在你的webapp里面添加一个contextloadlistener，在里面销毁所有dubbo资源。dubbo服务放到webapp里最好把那些使用jvm hook的地方改掉，因为如果webapp跑在tomcat这类容器中的话webapp的生命周期和容器生命周期不一致，webapp反部署时hook无法被触发就会造成资源泄漏。
@jiakai0419

@gMan1990
Copy link

gMan1990 commented Mar 1, 2017 •
edited

It works!
thanks @wuwen5 .

怎么用？ @jiakai0419

跑junit经常有Worker has already been shutdown
@ContextConfiguration("classpath:dubbo.xml")

@wuwen5 wuwen5 mentioned this issue Aug 24, 2017
Closed
@chickenlj chickenlj added this to the 2.5.4.M1 milestone Aug 24, 2017
@chickenlj chickenlj reopened this Aug 24, 2017
@diecui1202 diecui1202 self-assigned this Aug 25, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
2.5.4.M1
Development

No branches or pull requests
6 participants
@diecui1202 @wuwen5 @gMan1990 @jiakai0419 @AnyinSong @chickenlj