[alibaba/arthas]Java 进程使用 cloudflare zlib 后，arthas 启动报 java.lang.NoClassDefFoundError: com/taobao/arthas/core/advisor/SpyInterceptors$SpyInterceptor1

环境信息

arthas-boot.jar 或者 as.sh 的版本： xxx
Arthas 版本: 3.6.7
操作系统版本: 4.19.91-009.ali4000.alios7.x86_64
目标进程的JVM版本: openjdk version "11.0.16.15-AJDKfp1" 2022-09-28
执行arthas-boot的版本:3.6.7

重现问题的步骤

参考：https://github.com/aws/aws-graviton-getting-started#zlib-on-linux 构建对应的 libz.so

在 JVM 启动参数时配置：

export LD_PRELOAD="${APP_HOME}/libz-1.2.8.so"
export LD_LIBRARY_PATH=${APP_HOME}:${LD_LIBRARY_PATH}
echo LD_PRELOAD=$LD_PRELOAD
echo LD_LIBRARY_PATH=$LD_LIBRARY_PATH

启动 Java 进程
通过 arthas 挂载观测结果，如执行 watch 命令

期望的结果

能够正常获取结果

实际运行的结果

java.lang.NoClassDefFoundError: com/taobao/arthas/core/advisor/SpyInterceptors$SpyInterceptor1
        at com.taobao.amazon.agent.AgentClientHandler.channelRead(AgentClientHandler.java)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:336)
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:308)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357)
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379)
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365)
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
        at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
        at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
        at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
        at java.base/java.lang.Thread.run(Thread.java:991)

yinwoods

补充说明：移除 LD_PRELOAD/LD_LIBRARY_PATH 后，arthas 能够正常工作

yinwoods

用jad反编译下，发现arthas插入的代码SpyInterceptors.SpyInterceptor1.atEnter 没有被 inline 掉，如果被inline掉的话，正常应该是 SpyAPI.atEnter：

[arthas@200263]$ jad com.taobao.amazon.agent.AgentClientHandler channelRead

ClassLoader:
+-jdk.internal.loader.ClassLoaders$AppClassLoader@5f4da5c3
  +-jdk.internal.loader.ClassLoaders$PlatformClassLoader@4426d594

       /*
        * WARNING - void declaration
        */
       @Override
       public void channelRead(ChannelHandlerContext channelHandlerContext, Object object) throws Exception {
           SpyInterceptors.SpyInterceptor1.atEnter((Object)this, AgentClientHandler.class, (String)"channelRead|(Lio/netty/channel/ChannelHandlerContext;Ljava/lang/Object;)V", (Object[])new Object[]{channelHandlerContext, object});
           try {

检查 arthas 日志，有下面的错误信息：

2022-11-30 11:35:19 [arthas-command-execute] ERROR c.t.arthas.core.advisor.Enhancer -enhancer error, class: com/taobao/amazon/agent/AgentClientHandler, method: channelRead, in
terceptor: com.alibaba.bytekit.asm.interceptor.InterceptorProcessor
java.lang.IllegalArgumentException: Unsupported class file major version 21845
        at com.alibaba.deps.org.objectweb.asm.ClassReader.<init>(ClassReader.java:199)
        at com.alibaba.deps.org.objectweb.asm.ClassReader.<init>(ClassReader.java:180)
        at com.alibaba.deps.org.objectweb.asm.ClassReader.<init>(ClassReader.java:166)
        at com.alibaba.deps.org.objectweb.asm.ClassReader.<init>(ClassReader.java:287)
        at com.alibaba.bytekit.utils.AsmUtils.loadClass(AsmUtils.java:47)
        at com.alibaba.bytekit.asm.interceptor.InterceptorProcessor.process(InterceptorProcessor.java:172)
        at com.taobao.arthas.core.advisor.Enhancer.transform(Enhancer.java:228)
        at com.taobao.arthas.core.advisor.TransformerManager$1.transform(TransformerManager.java:51)
        at java.instrument/java.lang.instrument.ClassFileTransformer.transform(ClassFileTransformer.java:246)
        at java.instrument/sun.instrument.TransformerManager.transform(TransformerManager.java:188)
        at java.instrument/sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:563)
        at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
        at java.instrument/sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:167)
        at com.taobao.arthas.core.util.InstrumentationUtils.retransformClasses(InstrumentationUtils.java:32)
        at com.taobao.arthas.core.command.klass100.JadCommand.processExactMatch(JadCommand.java:177)
        at com.taobao.arthas.core.command.klass100.JadCommand.process(JadCommand.java:158)
        at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.process(AnnotatedCommandImpl.java:82)
        at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl.access$100(AnnotatedCommandImpl.java:18)
        at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:111)
        at com.taobao.arthas.core.shell.command.impl.AnnotatedCommandImpl$ProcessHandler.handle(AnnotatedCommandImpl.java:108)
        at com.taobao.arthas.core.shell.system.impl.ProcessImpl$CommandProcessTask.run(ProcessImpl.java:385)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:991)

所以是 arthas在做增强时，增强到一半，也就是执行inline时出错了，但是并没有把增强的结果去掉，只是简单打印了异常。

https://github.com/alibaba/arthas/blob/arthas-all-3.6.7/core/src/main/java/com/taobao/arthas/core/advisor/Enhancer.java#L240

那么为什么会出错呢？

参考这个： https://github.com/alibaba/arthas/issues/2319 ， asm官方认为是上游的代码有问题。从关联的issue来看，可能高版本的jdk修复了这个问题：

https://github.com/openjdk/jdk/pull/7492

当然，arthas后续应该优化这个 Enhancer 的逻辑，避免只增强一半的情况。

hengyunabc

[alibaba/arthas]Java 进程使用 cloudflare zlib 后，arthas 启动报 java.lang.NoClassDefFoundError: com/taobao/arthas/core/advisor/SpyInterceptors$SpyInterceptor1

回答