[apache/dubbo]重启tomcat的时候DubboShutdownHook会报一个非常讨厌的WARN

主动自行调用ProtocolConfig.destroyAll();，保障在jvm退出之前调用。 这样当DubboShutdownHook再次执行的时候待销毁资源已经clear了，就不会有异常了。

看起来像dubbo执行shutdown hook的时候，web application已经停掉了。 说明dubbo服务没有优雅停机.

按照上面的方法修改后，报了下面的WARN。

2016-10-18 21:30:54.854  WARN localhost-startStop-2 RegistryProtocol:132 -  [DUBBO] Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797399783&version=1.0.0 to registry 172.21.134.7:2005, cause: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797399783&version=1.0.0 to zookeeper zookeeper://172.21.134.7:2005/com.alibaba.dubbo.registry.RegistryService?application=stormsun&dubbo=2.8.4&interface=com.alibaba.dubbo.registry.RegistryService&owner=kaijia&pid=6586&timestamp=1476797399775, cause: null, dubbo version: 2.8.4, current host: 127.0.0.1
java.lang.IllegalStateException: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797399783&version=1.0.0 to registry 172.21.134.7:2005, cause: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797399783&version=1.0.0 to zookeeper zookeeper://172.21.134.7:2005/com.alibaba.dubbo.registry.RegistryService?application=stormsun&dubbo=2.8.4&interface=com.alibaba.dubbo.registry.RegistryService&owner=kaijia&pid=6586&timestamp=1476797399775, cause: null
    at com.alibaba.dubbo.registry.support.FailbackRegistry.unregister(FailbackRegistry.java:173)
    at com.alibaba.dubbo.registry.integration.RegistryProtocol$1.unexport(RegistryProtocol.java:130)
    at com.alibaba.dubbo.config.ServiceConfig.unexport(ServiceConfig.java:271)
    at com.alibaba.dubbo.config.spring.AnnotationBean.destroy(AnnotationBean.java:120)
    at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:261)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:972)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:979)
    at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1000)
    at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:976)
    at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:928)
    at org.springframework.web.context.ContextLoader.closeWebApplicationContext(ContextLoader.java:583)
    at org.springframework.web.context.ContextLoaderListener.contextDestroyed(ContextLoaderListener.java:116)
    at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4858)
    at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5474)
    at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:224)
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1424)
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1413)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.alibaba.dubbo.rpc.RpcException: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797399783&version=1.0.0 to zookeeper zookeeper://172.21.134.7:2005/com.alibaba.dubbo.registry.RegistryService?application=stormsun&dubbo=2.8.4&interface=com.alibaba.dubbo.registry.RegistryService&owner=kaijia&pid=6586&timestamp=1476797399775, cause: null
    at com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry.doUnregister(ZookeeperRegistry.java:108)
    at com.alibaba.dubbo.registry.support.FailbackRegistry.unregister(FailbackRegistry.java:160)
    ... 23 more
Caused by: java.lang.NullPointerException
    at org.I0Itec.zkclient.ZkClient$8.call(ZkClient.java:720)
    at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
    at org.I0Itec.zkclient.ZkClient.delete(ZkClient.java:716)
    at com.alibaba.dubbo.remoting.zookeeper.zkclient.ZkclientZookeeperClient.delete(ZkclientZookeeperClient.java:61)
    at com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry.doUnregister(ZookeeperRegistry.java:106)
    ... 24 more
2016-10-18 21:30:54.855  INFO localhost-startStop-2 ZookeeperRegistry:343 -  [DUBBO] Unsubscribe: provider://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&category=configurators&check=false&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797399783&version=1.0.0, dubbo version: 2.8.4, current host: 127.0.0.1
2016-10-18 21:30:54.856  INFO localhost-startStop-2 ZookeeperRegistry:312 -  [DUBBO] Unregister: dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.open.OpenApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.open.OpenApi&methods=queryFriendInfo,queryChatroomInfo,alterChatroomNotice,alterSelfDisplayName,batchSend,alterChatroomName,alterEntityPriority,queryChatrommMemberCount,liberate,applyXRobot,sendImage,sendText,kickSomeBody,manage,fire,sendVoice,transferOwner,send,queryUsernameChatroomInfo&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797400203&version=1.0.0, dubbo version: 2.8.4, current host: 127.0.0.1
2016-10-18 21:30:54.856  WARN localhost-startStop-2 RegistryProtocol:132 -  [DUBBO] Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.open.OpenApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.open.OpenApi&methods=queryFriendInfo,queryChatroomInfo,alterChatroomNotice,alterSelfDisplayName,batchSend,alterChatroomName,alterEntityPriority,queryChatrommMemberCount,liberate,applyXRobot,sendImage,sendText,kickSomeBody,manage,fire,sendVoice,transferOwner,send,queryUsernameChatroomInfo&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797400203&version=1.0.0 to registry 172.21.134.7:2005, cause: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.open.OpenApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.open.OpenApi&methods=queryFriendInfo,queryChatroomInfo,alterChatroomNotice,alterSelfDisplayName,batchSend,alterChatroomName,alterEntityPriority,queryChatrommMemberCount,liberate,applyXRobot,sendImage,sendText,kickSomeBody,manage,fire,sendVoice,transferOwner,send,queryUsernameChatroomInfo&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797400203&version=1.0.0 to zookeeper zookeeper://172.21.134.7:2005/com.alibaba.dubbo.registry.RegistryService?application=stormsun&dubbo=2.8.4&interface=com.alibaba.dubbo.registry.RegistryService&owner=kaijia&pid=6586&timestamp=1476797399775, cause: null, dubbo version: 2.8.4, current host: 127.0.0.1
java.lang.IllegalStateException: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.open.OpenApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.open.OpenApi&methods=queryFriendInfo,queryChatroomInfo,alterChatroomNotice,alterSelfDisplayName,batchSend,alterChatroomName,alterEntityPriority,queryChatrommMemberCount,liberate,applyXRobot,sendImage,sendText,kickSomeBody,manage,fire,sendVoice,transferOwner,send,queryUsernameChatroomInfo&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797400203&version=1.0.0 to registry 172.21.134.7:2005, cause: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.open.OpenApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.open.OpenApi&methods=queryFriendInfo,queryChatroomInfo,alterChatroomNotice,alterSelfDisplayName,batchSend,alterChatroomName,alterEntityPriority,queryChatrommMemberCount,liberate,applyXRobot,sendImage,sendText,kickSomeBody,manage,fire,sendVoice,transferOwner,send,queryUsernameChatroomInfo&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797400203&version=1.0.0 to zookeeper zookeeper://172.21.134.7:2005/com.alibaba.dubbo.registry.RegistryService?application=stormsun&dubbo=2.8.4&interface=com.alibaba.dubbo.registry.RegistryService&owner=kaijia&pid=6586&timestamp=1476797399775, cause: null
    at com.alibaba.dubbo.registry.support.FailbackRegistry.unregister(FailbackRegistry.java:173)
    at com.alibaba.dubbo.registry.integration.RegistryProtocol$1.unexport(RegistryProtocol.java:130)
    at com.alibaba.dubbo.config.ServiceConfig.unexport(ServiceConfig.java:271)
    at com.alibaba.dubbo.config.spring.AnnotationBean.destroy(AnnotationBean.java:120)
    at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:261)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:972)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:979)
    at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1000)
    at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:976)
    at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:928)
    at org.springframework.web.context.ContextLoader.closeWebApplicationContext(ContextLoader.java:583)
    at org.springframework.web.context.ContextLoaderListener.contextDestroyed(ContextLoaderListener.java:116)
    at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4858)
    at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5474)
    at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:224)
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1424)
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1413)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.alibaba.dubbo.rpc.RpcException: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.open.OpenApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.open.OpenApi&methods=queryFriendInfo,queryChatroomInfo,alterChatroomNotice,alterSelfDisplayName,batchSend,alterChatroomName,alterEntityPriority,queryChatrommMemberCount,liberate,applyXRobot,sendImage,sendText,kickSomeBody,manage,fire,sendVoice,transferOwner,send,queryUsernameChatroomInfo&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797400203&version=1.0.0 to zookeeper zookeeper://172.21.134.7:2005/com.alibaba.dubbo.registry.RegistryService?application=stormsun&dubbo=2.8.4&interface=com.alibaba.dubbo.registry.RegistryService&owner=kaijia&pid=6586&timestamp=1476797399775, cause: null
    at com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry.doUnregister(ZookeeperRegistry.java:108)
    at com.alibaba.dubbo.registry.support.FailbackRegistry.unregister(FailbackRegistry.java:160)
    ... 23 more
Caused by: java.lang.NullPointerException
    at org.I0Itec.zkclient.ZkClient$8.call(ZkClient.java:720)
    at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:675)
    at org.I0Itec.zkclient.ZkClient.delete(ZkClient.java:716)
    at com.alibaba.dubbo.remoting.zookeeper.zkclient.ZkclientZookeeperClient.delete(ZkclientZookeeperClient.java:61)
    at com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry.doUnregister(ZookeeperRegistry.java:106)
    ... 24 more
2016-10-18 21:30:54.856  INFO localhost-startStop-2 ZookeeperRegistry:343 -  [DUBBO] Unsubscribe: provider://172.21.134.7:20883/com.baijia.storm.sun.api.open.OpenApi?anyhost=true&application=stormsun&category=configurators&check=false&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.open.OpenApi&methods=queryFriendInfo,queryChatroomInfo,alterChatroomNotice,alterSelfDisplayName,batchSend,alterChatroomName,alterEntityPriority,queryChatrommMemberCount,liberate,applyXRobot,sendImage,sendText,kickSomeBody,manage,fire,sendVoice,transferOwner,send,queryUsernameChatroomInfo&owner=kaijia&pid=6586&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1476797400203&version=1.0.0, dubbo version: 2.8.4, current host: 127.0.0.1
2016-10-18 21:30:54.890 ERROR localhost-startStop-1-SendThread(172.21.134.7:2005) ClientCnxn:418 - from localhost-startStop-1-SendThread(172.21.134.7:2005)
java.lang.NoClassDefFoundError: org/apache/zookeeper/server/ZooTrace
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1220)
Caused by: java.lang.ClassNotFoundException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1315)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1178)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1139)
    ... 1 more
Caused by: java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1325)
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1313)
    ... 3 more
2016-10-18 21:30:54.938  INFO DubboShutdownHook AbstractConfig:450 -  [DUBBO] Run shutdown hook now., dubbo version: 2.8.4, current host: 127.0.0.1
2016-10-18 21:30:54.938  INFO DubboShutdownHook AbstractRegistryFactory:63 -  [DUBBO] Close all registries [], dubbo version: 2.8.4, current host: 127.0.0.1
2016-10-18 21:30:59.305  INFO localhost-startStop-1 ContextLoader:307 - Root WebApplicationContext: initialization started

https://github.com/alibaba/dubbo/issues/95 这个issue说原因是消费端一直没有在ZooKeeper注册中心注销 说要改zookeeper版本问题, 我试了下没什么用。

           <dependency>
                <groupId>com.github.sgroschupf</groupId>
                <artifactId>zkclient</artifactId>
                <version>0.1</version>
                <exclusions>
                    <exclusion>
                        <groupId>org.apache.zookeeper</groupId>
                        <artifactId>zookeeper</artifactId>
                    </exclusion>
                </exclusions>
            </dependency>
            <dependency>
                <groupId>org.apache.zookeeper</groupId>
                <artifactId>zookeeper</artifactId>
                <version>3.4.5</version>
            </dependency>

这个是因为ProtocolConfig.destroyAll()的时候，资源已经释放，zkclient连接已经关闭，你使用的应该是0.1版本的zkclient，close的时候_connection会置为null， 接着spring容器close的时候，dubbo服务会再次执行释放操作，最终会再次调用到zkclient的 delete方法，报NPE异常。

可以升级zkclient版本到0.5以上，最新版本是0.9，我们目前是0.6.

zkclient 修复记录 https://github.com/sgroschupf/zkclient/commit/0630c9c6e67ab49a51e80bfd939e4a0d01a69dfe

注：是group为com.101tec

更新了zkclient的version, null pointer变成了更清晰的解释ZkClient already closed，针对这个warning我是可以接受的，但是下面报了zk的error看起来像没有优雅的结束zk，这个怎么搞定？

            <dependency>
                <groupId>com.101tec</groupId>
                <artifactId>zkclient</artifactId>
                <version>0.8</version>
            </dependency>

2016-10-21 13:17:21.530  WARN localhost-startStop-2 RegistryProtocol:132 -  [DUBBO] Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=2434&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1477026966843&version=1.0.0 to registry 172.21.134.7:2005, cause: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=2434&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1477026966843&version=1.0.0 to zookeeper zookeeper://172.21.134.7:2005/com.alibaba.dubbo.registry.RegistryService?application=stormsun&dubbo=2.8.4&interface=com.alibaba.dubbo.registry.RegistryService&owner=kaijia&pid=2434&timestamp=1477026966835, cause: ZkClient already closed!, dubbo version: 2.8.4, current host: 127.0.0.1
java.lang.IllegalStateException: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=2434&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1477026966843&version=1.0.0 to registry 172.21.134.7:2005, cause: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=2434&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1477026966843&version=1.0.0 to zookeeper zookeeper://172.21.134.7:2005/com.alibaba.dubbo.registry.RegistryService?application=stormsun&dubbo=2.8.4&interface=com.alibaba.dubbo.registry.RegistryService&owner=kaijia&pid=2434&timestamp=1477026966835, cause: ZkClient already closed!
    at com.alibaba.dubbo.registry.support.FailbackRegistry.unregister(FailbackRegistry.java:173)
    at com.alibaba.dubbo.registry.integration.RegistryProtocol$1.unexport(RegistryProtocol.java:130)
    at com.alibaba.dubbo.config.ServiceConfig.unexport(ServiceConfig.java:271)
    at com.alibaba.dubbo.config.spring.AnnotationBean.destroy(AnnotationBean.java:120)
    at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:261)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:578)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:554)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingleton(DefaultListableBeanFactory.java:972)
    at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:523)
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.destroySingletons(DefaultListableBeanFactory.java:979)
    at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1000)
    at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:976)
    at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:928)
    at org.springframework.web.context.ContextLoader.closeWebApplicationContext(ContextLoader.java:583)
    at org.springframework.web.context.ContextLoaderListener.contextDestroyed(ContextLoaderListener.java:116)
    at org.apache.catalina.core.StandardContext.listenerStop(StandardContext.java:4858)
    at org.apache.catalina.core.StandardContext.stopInternal(StandardContext.java:5474)
    at org.apache.catalina.util.LifecycleBase.stop(LifecycleBase.java:224)
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1424)
    at org.apache.catalina.core.ContainerBase$StopChild.call(ContainerBase.java:1413)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.alibaba.dubbo.rpc.RpcException: Failed to unregister dubbo://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=2434&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1477026966843&version=1.0.0 to zookeeper zookeeper://172.21.134.7:2005/com.alibaba.dubbo.registry.RegistryService?application=stormsun&dubbo=2.8.4&interface=com.alibaba.dubbo.registry.RegistryService&owner=kaijia&pid=2434&timestamp=1477026966835, cause: ZkClient already closed!
    at com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry.doUnregister(ZookeeperRegistry.java:108)
    at com.alibaba.dubbo.registry.support.FailbackRegistry.unregister(FailbackRegistry.java:160)
    ... 23 more
Caused by: java.lang.IllegalStateException: ZkClient already closed!
    at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:987)
    at org.I0Itec.zkclient.ZkClient.delete(ZkClient.java:1047)
    at org.I0Itec.zkclient.ZkClient.delete(ZkClient.java:1042)
    at com.alibaba.dubbo.remoting.zookeeper.zkclient.ZkclientZookeeperClient.delete(ZkclientZookeeperClient.java:61)
    at com.alibaba.dubbo.registry.zookeeper.ZookeeperRegistry.doUnregister(ZookeeperRegistry.java:106)
    ... 24 more
2016-10-21 13:17:21.531  INFO localhost-startStop-2 ZookeeperRegistry:343 -  [DUBBO] Unsubscribe: provider://172.21.134.7:20883/com.baijia.storm.sun.api.robot.RobotApi?anyhost=true&application=stormsun&category=configurators&check=false&dubbo=2.8.4&generic=false&interface=com.baijia.storm.sun.api.robot.RobotApi&methods=writePrismRecord,updateUsernameChatroomRel,updateWeChatFriend,updateQrCodeChatroomRel,feedbackTask,pickTask,feedbackTaskV2,updateWeChatroom&owner=kaijia&pid=2434&revision=1.0.0-SNAPSHOT&side=provider&timestamp=1477026966843&version=1.0.0, dubbo version: 2.8.4, current host: 127.0.0.1
2016-10-21 13:17:21.618 ERROR localhost-startStop-1-SendThread(172.21.134.7:2005) ClientCnxn:414 - from localhost-startStop-1-SendThread(172.21.134.7:2005)
java.lang.NoClassDefFoundError: org/apache/zookeeper/server/ZooTrace
    at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1128)
Caused by: java.lang.ClassNotFoundException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1315)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1178)
    at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1139)
    ... 1 more
Caused by: java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [org.apache.zookeeper.server.ZooTrace]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1325)
    at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1313)
    ... 3 more
2016-10-21 13:17:21.679  INFO DubboShutdownHook AbstractConfig:450 -  [DUBBO] Run shutdown hook now., dubbo version: 2.8.4, current host: 127.0.0.1
2016-10-21 13:17:21.679  INFO DubboShutdownHook AbstractRegistryFactory:63 -  [DUBBO] Close all registries [], dubbo version: 2.8.4, current host: 127.0.0.1
2016-10-21 13:17:25.558  INFO localhost-startStop-1 ContextLoader:307 - Root WebApplicationContext: initialization started

ZooTrace 视乎仅仅只是个打日志的，不影响。 因为zookeeper client端 这段代码从来没有执行过，只在最后释放的时候打了个日志，所以这个类只有在释放的时候才会被加载，而tomcat容器已经停止不能再load class了。

这个异常可以尝试 规避下。 在tomcat停止之前，你在项目代码中加上一段（保障能执行,例如 写在ProtocolConfig.destroyAll()代码的位置）或者也可以在启动的地方加一下.

   ZooTrace.logTraceMessage(LOG, ZooTrace.getTextTraceLevel(),
                                     "Run shutdown now.");

另外 你再检查下是否有zookeeper包版本冲突.

报了两个新的异常，猜测是关闭dubbo的时候他想把现有的任务执行完毕，但是依赖的资源已经销毁了。 我有什么办法能控制销毁顺序吗，让这些异常也避免掉？

我目前是通过这种方法调用destoryAll()的。

@Slf4j
@Component
public class GraceShutdownListener {
    @PreDestroy
    public void graceShutdown() {
        log.info("GRACE SHUTDOWN");
        ZooTrace.logTraceMessage(log, ZooTrace.getTextTraceLevel(), "manual ZooTrace");
        ProtocolConfig.destroyAll();
    }
}

2016-10-21 14:44:08.240 ERROR DubboServerHandler-172.21.134.7:20883-thread-17 OpenApiImpl:97 - SYS_ERROR sunChat[{"weChatMsgType":3,"content":"http://file.gsxservice.com/stormEarth/robotFiles/imgMsg/6801998040@chatroom/342311477032243354.png","addressList":["7815031688@chatroom","7757035007@chatroom","7984035562@chatroom","8188035593@chatroom","7815031688@chatroom","7882043131@chatroom","7905043678@chatroom","8061042899@chatroom","7963049048@chatroom","6778991480@chatroom","6610991914@chatroom","7057992665@chatroom"],"addressType":1}]
redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool

2016-10-21 14:44:08.248  WARN DubboServerHandler-172.21.134.7:20883-thread-17 ChannelEventRunnable:84 -  [DUBBO] ChannelEventRunnable handle RECEIVED operation error, channel is NettyChannel [channel=[id: 0xc3355077, /172.21.134.7:34208 :> /172.21.134.7:20883]], message is Request [id=2660, version=2.0.0, twoway=true, event=false, broken=false, data=RpcInvocation [methodName=send, parameterTypes=[class com.baijia.storm.sun.api.common.proto.SunChat], arguments=[SunChat(weChatMsgType=3, content=http://file.gsxservice.com/stormEarth/robotFiles/imgMsg/6801998040@chatroom/342311477032243354.png, addressList=[7815031688@chatroom, 7757035007@chatroom, 7984035562@chatroom, 8188035593@chatroom, 7815031688@chatroom, 7882043131@chatroom, 7905043678@chatroom, 8061042899@chatroom, 7963049048@chatroom, 6778991480@chatroom, 6610991914@chatroom, 7057992665@chatroom], addressType=1)], attachments={path=com.baijia.storm.sun.api.open.OpenApi, input=697, dubbo=2.8.4, interface=com.baijia.storm.sun.api.open.OpenApi, version=1.0.0}]], dubbo version: 2.8.4, current host: 127.0.0.1
com.alibaba.dubbo.remoting.RemotingException: Failed to send message Response [id=2660, version=2.0.0, status=20, event=false, error=null, result=RpcResult [result=SunApiResponse(code=-3, msg=sys error, data=null), exception=null]] to /172.21.134.7:34208, cause: null
    at com.alibaba.dubbo.remoting.transport.netty.NettyChannel.send(NettyChannel.java:108)
    at com.alibaba.dubbo.remoting.transport.AbstractPeer.send(AbstractPeer.java:51)
    at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:171)
    at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
    at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.channels.ClosedChannelException
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:434)
    at org.jboss.netty.channel.socket.nio.AbstractNioWorker.writeFromUserCode(AbstractNioWorker.java:129)
    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.handleAcceptedSocket(NioServerSocketPipelineSink.java:99)
    at org.jboss.netty.channel.socket.nio.NioServerSocketPipelineSink.eventSunk(NioServerSocketPipelineSink.java:36)
    at org.jboss.netty.channel.Channels.write(Channels.java:725)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
    at org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
    at com.alibaba.dubbo.remoting.transport.netty.NettyHandler.writeRequested(NettyHandler.java:99)
    at org.jboss.netty.channel.Channels.write(Channels.java:704)
    at org.jboss.netty.channel.Channels.write(Channels.java:671)
    at org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:248)
    at com.alibaba.dubbo.remoting.transport.netty.NettyChannel.send(NettyChannel.java:98)
    ... 7 more

在你的graceShutdown方法中做下连接检测，保障所有已经进来的请求全部吐出去了。

例如

step 1. AbstractRegistryFactory.destroyAll() 断开注册中心的连接，这样可以保障没有新的请求进入这台负载。

step 2. 连接检测，可以通过dubbo filter 做一个全局的请求计数器，检测所有请求都出去了。 (这一步你可以先sleep 一段时间测试下。)

step 3. ProtocolConfig.destroyAll()

It works! thanks @wuwen5 .

@PreDestroy会在spring 容器destroy处理开始之前执行，你的redis连接池应该是由spring容器管理，spring容器会有序销毁资源。

最好不要在webapp中使用hook，webapp反部署时无法触发是一方面，更重要是的影响元空间回收

@AnyinSong 可以展开在说一下嘛。 例如想要在webapp中提供dubbo服务，关于grace shutdown有什么更具体的建议。

如果要在webapp中提供dubbo服务的话那在这个webapp的反部署逻辑中就要销毁所有dubbo资源，不能等到tomcat进程停止时做，比如在你的webapp里面添加一个contextloadlistener，在里面销毁所有dubbo资源。dubbo服务放到webapp里最好把那些使用jvm hook的地方改掉，因为如果webapp跑在tomcat这类容器中的话webapp的生命周期和容器生命周期不一致，webapp反部署时hook无法被触发就会造成资源泄漏。 @jiakai0419

It works! thanks @wuwen5 .

怎么用？ @jiakai0419

跑junit经常有Worker has already been shutdown @ContextConfiguration("classpath:dubbo.xml")