dubbo 反序列化空指针排查

背景

一个线上问题，dubbo provider 接收到的dto对象时，报空指针错误。但是将同样的dto第二次调用，就没问题。异常如下

java.lang.NullPointerException: null
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at com.alibaba.com.caucho.hessian.io.JavaDeserializer.instantiate(JavaDeserializer.java:305)
        at com.alibaba.com.caucho.hessian.io.JavaDeserializer.readObject(JavaDeserializer.java:194)
        at com.alibaba.com.caucho.hessian.io.SerializerFactory.readObject(SerializerFactory.java:527)
        at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObjectInstance(Hessian2Input.java:2743)
        at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2683)
        at com.alibaba.com.caucho.hessian.io.Hessian2Input.readObject(Hessian2Input.java:2657)
        at com.alibaba.dubbo.common.serialize.support.hessian.Hessian2ObjectInput.readObject(Hessian2ObjectInput.java:74)
        at com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcResult.decode(DecodeableRpcResult.java:90)
        at com.alibaba.dubbo.rpc.protocol.dubbo.DecodeableRpcResult.decode(DecodeableRpcResult.java:110)
        at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCodec.decodeBody(DubboCodec.java:88)
        at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.decode(ExchangeCodec.java:121)
        at com.alibaba.dubbo.remoting.exchange.codec.ExchangeCodec.decode(ExchangeCodec.java:82)
        at com.alibaba.dubbo.rpc.protocol.dubbo.DubboCountCodec.decode(DubboCountCodec.java:44)
        at com.alibaba.dubbo.remoting.transport.netty.NettyCodecAdapter$InternalDecoder.messageReceived(NettyCodecAdapter.java:133)
        at org.jboss.netty.channel.SimpleChannelUpstreamHandler.handleUpstream(SimpleChannelUpstreamHandler.java:80)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:564)
        at org.jboss.netty.channel.DefaultChannelPipeline.sendUpstream(DefaultChannelPipeline.java:559)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:274)
        at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:261)
        at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:349)
        at org.jboss.netty.channel.socket.nio.NioWorker.processSelectedKeys(NioWorker.java:280)
        at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:200)
        at org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.java:108)
        at org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWorker.java:44)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

原因

1、 dubbo 服务端因为某些原因报空指针异常
2、 java 虚拟机因为当某些异常频发时，会对此异常优化，不再打印堆栈，提升虚拟机效率，如下：

2020-01-12 19:01:15.999 [dubbo:4b69fcc5e21c4a1587d0512be618d2fb-1578826875996] ERROR DubboLogFilter - dubbo请求异常
java.lang.NullPointerException: null

1、 服务端将异常正常序列化返回消费端，但是消费端解析失败
3.1 消费端获取服务端异常，开始解析

3.2 打印异常堆栈信息

3.3 发现异常堆栈信息为空，此时发生误判。误判为当前方法发生空指针，结果返回当前方法的堆栈信息。

解决

1、 解决业务的空指针异常
2、 问题根源就出在服务提供端抛了大量NPE异常, 被JVM优化后, 没有异常栈了, 导致后续调用端复用错了异常栈, 解决办法是在服务提供端加上一个JVM参数 -XX:-OmitStackTraceInFastThrow , 禁止JVM的这个优化
3、 升级dubbo版本到2.7.x 。 经验证，dubbo2.6.x系列都有这个问题

参考

参考

文章永久链接：https://tech.souyunku.com/25039