2025-04 应用内单台服务器无响应问题排查
一、问题现象
- 2025-04-09 18:16:35 业务报演出票务后台访问异常、无法访问
- 2025-04-09 18:19:56 通过 CI/CD 平台发现 xx.xxx.98.69 这台机器异常，请求全部 500 应用无响应
二、止损手段
- 2025-04-09 18:22:34 通过 CI/CD 平台将 xx.xxx.98.69 这台异常机器剔除负载后应用访问正常
三、排查过程
1. 错误排查方向
一开始怀疑内存泄漏，查看机器上的 GC 日志，没看到有 GC 日志文件，现在看是找错路径了，常规都会写入到 /data/logs/skynet-marketcenter.java.all.ticket.manager.service/* 这个应用对应的日志目录下，但现在看 JVM 参数是直接写入到了 /data/logs/* 的目录下。
cat /data/logs/marketcenter.java.all.ticket.manager.service_gc.log.0.current然后尝试使用 Arthas dump 了 xx.xxx.98.69 这台异常机器内存，并选择了一台仍在正常接受请求的机器 dump 内存进行了对比。
无响应机器 dump 文件：https://tcstore1.17usoft.com/furt_file/furt--marketcenter-java-all-ticket-manager--10adc2f-1744195040446907302
正常的机器 dump 文件：https://tcstore1.17usoft.com/furt_file/furt--marketcenter-java-all-ticket-manager--10af89b-1744196202533725793
无响应机器二次 dump 文件：https://tcstore1.17usoft.com/furt_file/furt--marketcenter-java-all-ticket-manager--10adc2f-1744196776709105619
通过 MAT 对比查看，确实发现经过多次 GC 仍然有较多 ImageIO 相关对象驻留在内容中。通过分析运行日志可以发现，有较多大图片未能正常下载或上传，导致不会写入缓存从而每次任务运行均会重复拉取，怀疑和此有关。[1]
之后尝试在本地复现，调低堆空间大小并通过一个较大的猫眼资源图片循环执行，发现未出现无响应或宕机情况。通过 JConsole 和 JProfiler 本地观察也未见堆空间内存使用异常。
// JVM -Xms500m -Xmx500m
public static void main(String[] args) {
WormContentProcessService wormContentProcessService = new WormContentProcessService();
for (int i = 0; i < 100; i++) {
wormContentProcessService.convertImageURL("https://p0.pipi.cn/mediaplus/fantasy_perform_fe/0fa3344e15d5c2e3271fae12653c7b19cc9aa", null, null); // 图片大小16.6 MB
}
}
2. 正确排查方向
此时又回过头看 APM JVM 监控，看到问题时段（18:10 ）老年代就再也没有增长过了，且 Old GC 也没有触发过了。并且看线程状态情况，发现总线程数大量增长且大量线程停留在 Waiting 状态上。此时发现排查方向不对，不是堆空间内存使用问题，而是线程问题。
通过 curl -v xx.xxx.98.69:8868/allticketconfig/monitor.jsp 查看此时应用是否还能正确接受请求，发现还是无响应，确定应用长时间仍然未恢复。
然后通过 Arthas 的 thread 命令查看 xx.xxx.98.69 异常机器，可以看到确实大量线程此时还是 WAITING 状态，并且有 200 个 Tomcat 的处理 HTTP 请求线程（http-nio-8868-exec）也全都是 WAITING 状态。
Tomcat 默认工作线程数（maxThreads）是 200 个 Apache Tomcat 8 Configuration Reference
thread dashboard（此文件筛选了仅 WAITING 状态的线程）
Threads Total: 496, NEW: 0, RUNNABLE: 39, BLOCKED: 0, WAITING: 408, TIMED_WAITING: 49, TERMINATED: 0
ID NAME GROUP PRIORITY STATE %CPU DELTA_TIME TIME INTERRUPTED DAEMON
2 Reference Handler system 10 WAITING 0.0 0.000 0:53.107 false true
3 Finalizer system 8 WAITING 0.0 0.000 1:23.185 false true
246 Java2D Disposer system 10 WAITING 0.0 0.000 0:0.121 false true
23981 arthas-timer system 9 WAITING 0.0 0.000 0:0.000 false true
13 weak-ref-cleaner main 5 WAITING 0.0 0.000 0:0.031 false true
28 Catalina-utility-2 main 1 WAITING 0.0 0.000 0:8.338 false false
31 Thread-7 main 5 WAITING 0.0 0.000 7:2.388 false true
39 Druid-ConnectionPool-Create-651316602 main 5 WAITING 0.0 0.000 0:1.731 false true
42 Druid-ConnectionPool-Create-1371989877 main 5 WAITING 0.0 0.000 0:0.465 false true
44 statisInfoExecutor-pool-1 main 5 WAITING 0.0 0.000 0:0.746 false false
45 taskExecutor-pool-1 main 5 WAITING 0.0 0.000 0:7.415 false false
62 Druid-ConnectionPool-Create-140733684 main 5 WAITING 0.0 0.000 0:2.124 false true
79 Druid-ConnectionPool-Create-1114182596 main 5 WAITING 0.0 0.000 0:2.548 false true
81 statisInfoExecutor-pool-1 main 5 WAITING 0.0 0.000 0:0.727 false false
82 taskExecutor-pool-1 main 5 WAITING 0.0 0.000 0:7.083 false false
85 Druid-ConnectionPool-Create-1784029781 main 5 WAITING 0.0 0.000 0:0.604 false true
88 Druid-ConnectionPool-Create-523655718 main 5 WAITING 0.0 0.000 0:0.586 false true
90 statisInfoExecutor-pool-1 main 5 WAITING 0.0 0.000 0:0.578 false false
91 taskExecutor-pool-1 main 5 WAITING 0.0 0.000 0:7.044 false false
108 Druid-ConnectionPool-Create-948391605 main 5 WAITING 0.0 0.000 0:0.289 false true
125 Druid-ConnectionPool-Create-601932288 main 5 WAITING 0.0 0.000 0:0.036 false true
127 statisInfoExecutor-pool-1 main 5 WAITING 0.0 0.000 0:0.683 false false
128 taskExecutor-pool-1 main 5 WAITING 0.0 0.000 0:7.072 false false
135 ClientHouseKeepingService main 5 WAITING 0.0 0.000 0:4.677 false true
138 MQClientFactoryScheduledThread main 5 WAITING 0.0 0.000 0:16.181 false false
141 NettyClientWorkerThread_1 main 5 WAITING 0.0 0.000 0:3.376 false false
136 PullMessageService main 5 WAITING 0.0 0.000 0:1.165 false false
137 RebalanceService main 5 WAITING 0.0 0.000 0:3.753 false false
143 NettyClientWorkerThread_2 main 5 WAITING 0.0 0.000 0:12.212 false false
144 NettyClientWorkerThread_3 main 5 WAITING 0.0 0.000 0:1.958 false false
145 NettyClientPublicExecutor_1 main 5 WAITING 0.0 0.000 0:0.575 false false
146 NettyClientWorkerThread_4 main 5 WAITING 0.0 0.000 0:4.130 false false
147 NettyClientPublicExecutor_2 main 5 WAITING 0.0 0.000 0:0.620 false false
148 NettyClientPublicExecutor_3 main 5 WAITING 0.0 0.000 0:0.577 false false
149 NettyClientPublicExecutor_4 main 5 WAITING 0.0 0.000 0:0.570 false false
151 ClientHouseKeepingService main 5 WAITING 0.0 0.000 0:4.863 false true
154 MQClientFactoryScheduledThread main 5 WAITING 0.0 0.000 0:15.466 false false
157 NettyClientWorkerThread_1 main 5 WAITING 0.0 0.000 0:6.844 false false
152 PullMessageService main 5 WAITING 0.0 0.000 0:1.172 false false
153 RebalanceService main 5 WAITING 0.0 0.000 0:3.509 false false
158 NettyClientWorkerThread_2 main 5 WAITING 0.0 0.000 0:2.255 false false
159 NettyClientWorkerThread_3 main 5 WAITING 0.0 0.000 0:1.960 false false
160 NettyClientPublicExecutor_1 main 5 WAITING 0.0 0.000 0:0.559 false false
161 NettyClientWorkerThread_4 main 5 WAITING 0.0 0.000 0:9.673 false false
162 NettyClientPublicExecutor_2 main 5 WAITING 0.0 0.000 0:0.559 false false
163 NettyClientPublicExecutor_3 main 5 WAITING 0.0 0.000 0:0.544 false false
164 NettyClientPublicExecutor_4 main 5 WAITING 0.0 0.000 0:0.562 false false
165 PullMessageServiceScheduledThread main 5 WAITING 0.0 0.000 0:0.000 false false
168 PullMessageService main 5 WAITING 0.0 0.000 0:0.000 false false
175 MQTT Snd: xx.xxx.98.69 main 5 WAITING 0.0 0.000 0:0.274 false false
176 MQTT Call: xx.xxx.98.69 main 5 WAITING 0.0 0.000 0:0.152 false false
178 http-nio-8868-exec-1 main 5 WAITING 0.0 0.000 1:45.860 false true
179 http-nio-8868-exec-2 main 5 WAITING 0.0 0.000 1:49.646 false true
180 http-nio-8868-exec-3 main 5 WAITING 0.0 0.000 1:44.337 false true
181 http-nio-8868-exec-4 main 5 WAITING 0.0 0.000 1:46.519 false true
182 http-nio-8868-exec-5 main 5 WAITING 0.0 0.000 1:50.388 false true
183 http-nio-8868-exec-6 main 5 WAITING 0.0 0.000 1:46.352 false true
184 http-nio-8868-exec-7 main 5 WAITING 0.0 0.000 1:47.028 false true
185 http-nio-8868-exec-8 main 5 WAITING 0.0 0.000 1:46.498 false true
186 http-nio-8868-exec-9 main 5 WAITING 0.0 0.000 1:46.429 false true
187 http-nio-8868-exec-10 main 5 WAITING 0.0 0.000 1:51.719 false true
224 pool-18-thread-1 main 5 WAITING 0.0 0.000 0:7.551 false false
225 AsyncAppender-Worker-null main 5 WAITING 0.0 0.000 0:57.016 false true
355 MultiThreadedHttpConnectionManager cleanup main 5 WAITING 0.0 0.000 0:0.000 false true
437 maoYanOpenService-fullCoverageDBData1 main 5 WAITING 0.0 0.000 0:14.283 false true
438 maoYanOpenService-fullCoverageDBData2 main 5 WAITING 0.0 0.000 0:14.464 false true
439 maoYanOpenService-fullCoverageDBData3 main 5 WAITING 0.0 0.000 0:14.510 false true
440 maoYanOpenService-fullCoverageDBData4 main 5 WAITING 0.0 0.000 0:14.335 false true
441 maoYanOpenService-fullCoverageDBData5 main 5 WAITING 0.0 0.000 0:14.437 false true
442 maoYanOpenService-fullCoverageDBData6 main 5 WAITING 0.0 0.000 0:14.325 false true
443 maoYanOpenService-fullCoverageDBData7 main 5 WAITING 0.0 0.000 0:14.413 false true
444 maoYanOpenService-fullCoverageDBData8 main 5 WAITING 0.0 0.000 0:14.426 false true
445 maoYanOpenService-fullCoverageDBData9 main 5 WAITING 0.0 0.000 0:14.394 false true
446 maoYanOpenService-fullCoverageDBData10 main 5 WAITING 0.0 0.000 0:14.514 false true
447 maoYanOpenService-fullCoverageDBData11 main 5 WAITING 0.0 0.000 0:14.270 false true
448 maoYanOpenService-fullCoverageDBData12 main 5 WAITING 0.0 0.000 0:14.362 false true
449 maoYanOpenService-fullCoverageDBData13 main 5 WAITING 0.0 0.000 0:14.393 false true
450 maoYanOpenService-fullCoverageDBData14 main 5 WAITING 0.0 0.000 0:14.433 false true
451 maoYanOpenService-fullCoverageDBData15 main 5 WAITING 0.0 0.000 0:14.656 false true
452 maoYanOpenService-fullCoverageDBData16 main 5 WAITING 0.0 0.000 0:14.416 false true
453 maoYanOpenService-fullCoverageDBData17 main 5 WAITING 0.0 0.000 0:14.260 false true
454 maoYanOpenService-fullCoverageDBData18 main 5 WAITING 0.0 0.000 0:14.281 false true
455 maoYanOpenService-fullCoverageDBData19 main 5 WAITING 0.0 0.000 0:14.375 false true
456 maoYanOpenService-fullCoverageDBData20 main 5 WAITING 0.0 0.000 0:14.381 false true
460 new-mao-yan-product-full-coverage-process-se main 5 WAITING 0.0 0.000 7:1.059 false true
461 new-mao-yan-product-full-coverage-process-se main 5 WAITING 0.0 0.000 7:4.346 false true
462 new-mao-yan-product-full-coverage-process-se main 5 WAITING 0.0 0.000 7:0.674 false true
463 new-mao-yan-product-full-coverage-process-se main 5 WAITING 0.0 0.000 7:1.964 false true
464 new-mao-yan-product-full-coverage-process-se main 5 WAITING 0.0 0.000 7:3.977 false true
465 new-mao-yan-product-full-coverage-process-se main 5 WAITING 0.0 0.000 6:57.190 false true
466 new-mao-yan-product-full-coverage-process-se main 5 WAITING 0.0 0.000 7:2.319 false true
467 new-mao-yan-product-full-coverage-process-se main 5 WAITING 0.0 0.000 6:59.074 false true
468 new-mao-yan-product-full-coverage-process-se main 5 WAITING 0.0 0.000 6:58.540 false true
469 new-mao-yan-product-full-coverage-process-se main 5 WAITING 0.0 0.000 6:58.770 false true
470 new-mao-yan-product-full-coverage-process-se main 5 WAITING 0.0 0.000 7:0.632 false true
471 new-mao-yan-product-full-coverage-process-se main 5 WAITING 0.0 0.000 6:56.059 false true
472 new-mao-yan-product-full-coverage-process-se main 5 WAITING 0.0 0.000 7:1.359 false true
473 new-mao-yan-product-full-coverage-process-se main 5 WAITING 0.0 0.000 6:58.733 false true
474 new-mao-yan-product-full-coverage-process-se main 5 WAITING 0.0 0.000 7:0.772 false true
475 new-mao-yan-product-full-coverage-process-se main 5 WAITING 0.0 0.000 7:2.583 false true
476 new-mao-yan-product-full-coverage-process-se main 5 WAITING 0.0 0.000 7:0.608 false true
477 new-mao-yan-product-full-coverage-process-se main 5 WAITING 0.0 0.000 7:3.673 false true
478 new-mao-yan-product-full-coverage-process-se main 5 WAITING 0.0 0.000 6:54.872 false true
479 new-mao-yan-product-full-coverage-process-se main 5 WAITING 0.0 0.000 7:0.614 false true
482 wormContentProcessService-newbieConvertExecu main 5 WAITING 0.0 0.000 22:27.504 false true
483 wormContentProcessService-newbieConvertExecu main 5 WAITING 0.0 0.000 22:20.086 false true
484 wormContentProcessService-newbieConvertExecu main 5 WAITING 0.0 0.000 22:44.929 false true
485 wormContentProcessService-newbieConvertExecu main 5 WAITING 0.0 0.000 22:46.088 false true
486 wormContentProcessService-newbieConvertExecu main 5 WAITING 0.0 0.000 20:47.646 false true
487 wormContentProcessService-newbieConvertExecu main 5 WAITING 0.0 0.000 23:14.042 false true
488 wormContentProcessService-newbieConvertExecu main 5 WAITING 0.0 0.000 23:42.165 false true
489 wormContentProcessService-newbieConvertExecu main 5 WAITING 0.0 0.000 18:25.116 false true
490 wormContentProcessService-newbieConvertExecu main 5 WAITING 0.0 0.000 19:53.351 false true
491 wormContentProcessService-newbieConvertExecu main 5 WAITING 0.0 0.000 20:55.743 false true
492 wormContentProcessService-newbieConvertExecu main 5 WAITING 0.0 0.000 21:4.969 false true
493 wormContentProcessService-newbieConvertExecu main 5 WAITING 0.0 0.000 18:20.443 false true
494 wormContentProcessService-newbieConvertExecu main 5 WAITING 0.0 0.000 18:46.891 false true
495 wormContentProcessService-newbieConvertExecu main 5 WAITING 0.0 0.000 24:27.252 false true
496 wormContentProcessService-newbieConvertExecu main 5 WAITING 0.0 0.000 23:43.189 false true
497 wormContentProcessService-newbieConvertExecu main 5 WAITING 0.0 0.000 20:37.290 false true
498 wormContentProcessService-newbieConvertExecu main 5 WAITING 0.0 0.000 22:8.042 false true
499 wormContentProcessService-newbieConvertExecu main 5 WAITING 0.0 0.000 19:25.118 false true
500 wormContentProcessService-newbieConvertExecu main 5 WAITING 0.0 0.000 23:58.284 false true
501 wormContentProcessService-newbieConvertExecu main 5 WAITING 0.0 0.000 21:58.669 false true
515 logback-1 main 5 WAITING 0.0 0.000 0:0.011 false true
522 http-nio-8868-exec-11 main 5 WAITING 0.0 0.000 1:41.908 false true
544 ticket-stats1 main 5 WAITING 0.0 0.000 0:0.007 false true
545 logback-2 main 5 WAITING 0.0 0.000 0:0.009 false true
582 logback-3 main 5 WAITING 0.0 0.000 0:0.010 false true
598 logback-4 main 5 WAITING 0.0 0.000 0:0.008 false true
627 logback-5 main 5 WAITING 0.0 0.000 0:0.009 false true
638 product-sync-source-data-service1 main 5 WAITING 0.0 0.000 1:6.961 false true
640 product-sync-source-data-service3 main 5 WAITING 0.0 0.000 1:6.939 false true
641 product-sync-source-data-service4 main 5 WAITING 0.0 0.000 1:7.030 false true
642 product-sync-source-data-service5 main 5 WAITING 0.0 0.000 1:6.869 false true
643 product-sync-source-data-service6 main 5 WAITING 0.0 0.000 1:7.045 false true
644 product-sync-source-data-service7 main 5 WAITING 0.0 0.000 1:6.954 false true
645 product-sync-source-data-service8 main 5 WAITING 0.0 0.000 1:7.344 false true
646 product-sync-source-data-service9 main 5 WAITING 0.0 0.000 1:6.877 false true
647 product-sync-source-data-service10 main 5 WAITING 0.0 0.000 1:6.410 false true
648 product-sync-source-data-service11 main 5 WAITING 0.0 0.000 1:7.208 false true
649 product-sync-source-data-service12 main 5 WAITING 0.0 0.000 1:7.037 false true
650 product-sync-source-data-service13 main 5 WAITING 0.0 0.000 1:7.003 false true
651 product-sync-source-data-service14 main 5 WAITING 0.0 0.000 1:7.066 false true
652 product-sync-source-data-service15 main 5 WAITING 0.0 0.000 1:6.875 false true
653 product-sync-source-data-service16 main 5 WAITING 0.0 0.000 1:6.805 false true
654 product-sync-source-data-service17 main 5 WAITING 0.0 0.000 1:6.966 false true
655 product-sync-source-data-service18 main 5 WAITING 0.0 0.000 1:6.761 false true
656 product-sync-source-data-service19 main 5 WAITING 0.0 0.000 1:7.115 false true
657 product-sync-source-data-service20 main 5 WAITING 0.0 0.000 1:7.258 false true
659 logback-6 main 5 WAITING 0.0 0.000 0:0.008 false true
660 logback-7 main 5 WAITING 0.0 0.000 0:0.008 false true
665 logback-8 main 5 WAITING 0.0 0.000 0:0.008 false true
682 http-nio-8868-exec-12 main 5 WAITING 0.0 0.000 1:39.609 false true
742 ticket-stats2 main 5 WAITING 0.0 0.000 0:0.006 false true
757 http-nio-8868-exec-13 main 5 WAITING 0.0 0.000 1:44.334 false true
758 http-nio-8868-exec-14 main 5 WAITING 0.0 0.000 1:40.637 false true
767 http-nio-8868-exec-15 main 5 WAITING 0.0 0.000 1:44.056 false true
1003 http-nio-8868-exec-16 main 5 WAITING 0.0 0.000 1:42.315 false true
3663 region-full-coverage-process-service1 main 5 WAITING 0.0 0.000 0:0.808 false true
3664 region-full-coverage-process-service2 main 5 WAITING 0.0 0.000 0:0.800 false true
3665 region-full-coverage-process-service3 main 5 WAITING 0.0 0.000 0:0.815 false true
3666 region-full-coverage-process-service4 main 5 WAITING 0.0 0.000 0:0.784 false true
3667 region-full-coverage-process-service5 main 5 WAITING 0.0 0.000 0:0.810 false true
3668 region-full-coverage-process-service6 main 5 WAITING 0.0 0.000 0:0.795 false true
3669 region-full-coverage-process-service7 main 5 WAITING 0.0 0.000 0:0.807 false true
3670 region-full-coverage-process-service8 main 5 WAITING 0.0 0.000 0:0.798 false true
3671 region-full-coverage-process-service9 main 5 WAITING 0.0 0.000 0:0.813 false true
3672 region-full-coverage-process-service10 main 5 WAITING 0.0 0.000 0:0.816 false true
3673 region-full-coverage-process-service11 main 5 WAITING 0.0 0.000 0:0.797 false true
3674 region-full-coverage-process-service12 main 5 WAITING 0.0 0.000 0:0.777 false true
3675 region-full-coverage-process-service13 main 5 WAITING 0.0 0.000 0:0.792 false true
3676 region-full-coverage-process-service14 main 5 WAITING 0.0 0.000 0:0.792 false true
3677 region-full-coverage-process-service15 main 5 WAITING 0.0 0.000 0:0.828 false true
3678 region-full-coverage-process-service16 main 5 WAITING 0.0 0.000 0:0.780 false true
3679 region-full-coverage-process-service17 main 5 WAITING 0.0 0.000 0:0.790 false true
3680 region-full-coverage-process-service18 main 5 WAITING 0.0 0.000 0:0.800 false true
3681 region-full-coverage-process-service19 main 5 WAITING 0.0 0.000 0:0.812 false true
3682 region-full-coverage-process-service20 main 5 WAITING 0.0 0.000 0:0.818 false true
3684 maoYanWormService-fullCoverageDBData1 main 5 WAITING 0.0 0.000 0:3.942 false true
3685 maoYanWormService-fullCoverageDBData2 main 5 WAITING 0.0 0.000 0:4.194 false true
3686 maoYanWormService-fullCoverageDBData3 main 5 WAITING 0.0 0.000 0:3.106 false true
3687 maoYanWormService-fullCoverageDBData4 main 5 WAITING 0.0 0.000 0:3.659 false true
3688 maoYanWormService-fullCoverageDBData5 main 5 WAITING 0.0 0.000 0:3.401 false true
3689 maoYanWormService-fullCoverageDBData6 main 5 WAITING 0.0 0.000 0:4.168 false true
3690 maoYanWormService-fullCoverageDBData7 main 5 WAITING 0.0 0.000 0:3.505 false true
3691 maoYanWormService-fullCoverageDBData8 main 5 WAITING 0.0 0.000 0:3.056 false true
3692 maoYanWormService-fullCoverageDBData9 main 5 WAITING 0.0 0.000 0:3.567 false true
3693 maoYanWormService-fullCoverageDBData10 main 5 WAITING 0.0 0.000 0:4.291 false true
3694 maoYanWormService-fullCoverageDBData11 main 5 WAITING 0.0 0.000 0:4.167 false true
3695 maoYanWormService-fullCoverageDBData12 main 5 WAITING 0.0 0.000 0:4.011 false true
3696 maoYanWormService-fullCoverageDBData13 main 5 WAITING 0.0 0.000 0:4.118 false true
3697 maoYanWormService-fullCoverageDBData14 main 5 WAITING 0.0 0.000 0:4.096 false true
3698 maoYanWormService-fullCoverageDBData15 main 5 WAITING 0.0 0.000 0:3.966 false true
3699 maoYanWormService-fullCoverageDBData16 main 5 WAITING 0.0 0.000 0:4.219 false true
3700 maoYanWormService-fullCoverageDBData17 main 5 WAITING 0.0 0.000 0:3.608 false true
3701 maoYanWormService-fullCoverageDBData18 main 5 WAITING 0.0 0.000 0:4.333 false true
3702 maoYanWormService-fullCoverageDBData19 main 5 WAITING 0.0 0.000 0:3.009 false true
3703 maoYanWormService-fullCoverageDBData20 main 5 WAITING 0.0 0.000 0:4.012 false true
3715 mao-yan-product-full-coverage-process-servic main 5 WAITING 0.0 0.000 0:14.835 false true
3716 mao-yan-product-full-coverage-process-servic main 5 WAITING 0.0 0.000 0:16.149 false true
3717 mao-yan-product-full-coverage-process-servic main 5 WAITING 0.0 0.000 0:15.330 false true
3718 mao-yan-product-full-coverage-process-servic main 5 WAITING 0.0 0.000 0:15.858 false true
3719 mao-yan-product-full-coverage-process-servic main 5 WAITING 0.0 0.000 0:17.920 false true
3720 mao-yan-product-full-coverage-process-servic main 5 WAITING 0.0 0.000 0:15.318 false true
3721 mao-yan-product-full-coverage-process-servic main 5 WAITING 0.0 0.000 0:15.929 false true
3722 mao-yan-product-full-coverage-process-servic main 5 WAITING 0.0 0.000 0:16.542 false true
3723 mao-yan-product-full-coverage-process-servic main 5 WAITING 0.0 0.000 0:15.976 false true
3724 mao-yan-product-full-coverage-process-servic main 5 WAITING 0.0 0.000 0:16.158 false true
3725 mao-yan-product-full-coverage-process-servic main 5 WAITING 0.0 0.000 0:17.524 false true
3726 mao-yan-product-full-coverage-process-servic main 5 WAITING 0.0 0.000 0:16.430 false true
3727 mao-yan-product-full-coverage-process-servic main 5 WAITING 0.0 0.000 0:17.949 false true
3728 mao-yan-product-full-coverage-process-servic main 5 WAITING 0.0 0.000 0:16.890 false true
3729 mao-yan-product-full-coverage-process-servic main 5 WAITING 0.0 0.000 0:16.416 false true
3730 mao-yan-product-full-coverage-process-servic main 5 WAITING 0.0 0.000 0:17.618 false true
3731 mao-yan-product-full-coverage-process-servic main 5 WAITING 0.0 0.000 0:16.324 false true
3732 mao-yan-product-full-coverage-process-servic main 5 WAITING 0.0 0.000 0:15.636 false true
3733 mao-yan-product-full-coverage-process-servic main 5 WAITING 0.0 0.000 0:15.830 false true
3734 mao-yan-product-full-coverage-process-servic main 5 WAITING 0.0 0.000 0:16.772 false true
12387 ticket-stats3 main 5 WAITING 0.0 0.000 0:0.004 false true
13317 ticket-stats4 main 5 WAITING 0.0 0.000 0:0.005 false true
13451 ticket-stats5 main 5 WAITING 0.0 0.000 0:0.003 false true
23629 product-item-full-coverage-process main 5 WAITING 0.0 0.000 0:0.882 false false
23726 updateProjectLikeCount main 5 WAITING 0.0 0.000 0:1.094 false false
23731 http-nio-8868-exec-17 main 5 WAITING 0.0 0.000 0:0.005 false true
23732 http-nio-8868-exec-18 main 5 WAITING 0.0 0.000 0:0.004 false true
23733 http-nio-8868-exec-19 main 5 WAITING 0.0 0.000 0:0.009 false true
23734 http-nio-8868-exec-20 main 5 WAITING 0.0 0.000 0:0.032 false true
23735 http-nio-8868-exec-21 main 5 WAITING 0.0 0.000 0:0.022 false true
23737 http-nio-8868-exec-22 main 5 WAITING 0.0 0.000 0:0.007 false true
23738 http-nio-8868-exec-23 main 5 WAITING 0.0 0.000 0:0.018 false true
23739 http-nio-8868-exec-24 main 5 WAITING 0.0 0.000 0:0.011 false true
23740 http-nio-8868-exec-25 main 5 WAITING 0.0 0.000 0:0.052 false true
23741 http-nio-8868-exec-26 main 5 WAITING 0.0 0.000 0:0.010 false true
23742 http-nio-8868-exec-27 main 5 WAITING 0.0 0.000 0:0.020 false true
23743 http-nio-8868-exec-28 main 5 WAITING 0.0 0.000 0:0.022 false true
23744 http-nio-8868-exec-29 main 5 WAITING 0.0 0.000 0:0.027 false true
23745 http-nio-8868-exec-30 main 5 WAITING 0.0 0.000 0:0.036 false true
23746 http-nio-8868-exec-31 main 5 WAITING 0.0 0.000 0:0.002 false true
23747 http-nio-8868-exec-32 main 5 WAITING 0.0 0.000 0:0.006 false true
23748 http-nio-8868-exec-33 main 5 WAITING 0.0 0.000 0:0.018 false true
23749 http-nio-8868-exec-34 main 5 WAITING 0.0 0.000 0:0.035 false true
23750 http-nio-8868-exec-35 main 5 WAITING 0.0 0.000 0:0.047 false true
23752 http-nio-8868-exec-36 main 5 WAITING 0.0 0.000 0:0.026 false true
23753 http-nio-8868-exec-37 main 5 WAITING 0.0 0.000 0:0.019 false true
23754 http-nio-8868-exec-38 main 5 WAITING 0.0 0.000 0:0.017 false true
23755 http-nio-8868-exec-39 main 5 WAITING 0.0 0.000 0:0.001 false true
23756 http-nio-8868-exec-40 main 5 WAITING 0.0 0.000 0:0.009 false true
23757 http-nio-8868-exec-41 main 5 WAITING 0.0 0.000 0:0.008 false true
23758 http-nio-8868-exec-42 main 5 WAITING 0.0 0.000 0:0.010 false true
23759 http-nio-8868-exec-43 main 5 WAITING 0.0 0.000 0:0.004 false true
23760 http-nio-8868-exec-44 main 5 WAITING 0.0 0.000 0:0.072 false true
23761 http-nio-8868-exec-45 main 5 WAITING 0.0 0.000 0:0.025 false true
23763 http-nio-8868-exec-46 main 5 WAITING 0.0 0.000 0:0.036 false true
23764 http-nio-8868-exec-47 main 5 WAITING 0.0 0.000 0:0.001 false true
23765 http-nio-8868-exec-48 main 5 WAITING 0.0 0.000 0:0.005 false true
23766 http-nio-8868-exec-49 main 5 WAITING 0.0 0.000 0:0.002 false true
23767 http-nio-8868-exec-50 main 5 WAITING 0.0 0.000 0:0.001 false true
23768 http-nio-8868-exec-51 main 5 WAITING 0.0 0.000 0:0.002 false true
23769 http-nio-8868-exec-52 main 5 WAITING 0.0 0.000 0:0.004 false true
23770 http-nio-8868-exec-53 main 5 WAITING 0.0 0.000 0:0.008 false true
23771 http-nio-8868-exec-54 main 5 WAITING 0.0 0.000 0:0.048 false true
23772 http-nio-8868-exec-55 main 5 WAITING 0.0 0.000 0:0.037 false true
23773 http-nio-8868-exec-56 main 5 WAITING 0.0 0.000 0:0.035 false true
23775 http-nio-8868-exec-57 main 5 WAITING 0.0 0.000 0:0.055 false true
23777 http-nio-8868-exec-58 main 5 WAITING 0.0 0.000 0:0.082 false true
23779 http-nio-8868-exec-59 main 5 WAITING 0.0 0.000 0:0.007 false true
23780 http-nio-8868-exec-60 main 5 WAITING 0.0 0.000 0:0.044 false true
23782 http-nio-8868-exec-61 main 5 WAITING 0.0 0.000 0:0.076 false true
23784 http-nio-8868-exec-62 main 5 WAITING 0.0 0.000 0:0.018 false true
23785 http-nio-8868-exec-63 main 5 WAITING 0.0 0.000 0:0.034 false true
23786 http-nio-8868-exec-64 main 5 WAITING 0.0 0.000 0:0.048 false true
23787 http-nio-8868-exec-65 main 5 WAITING 0.0 0.000 0:0.031 false true
23789 http-nio-8868-exec-66 main 5 WAITING 0.0 0.000 0:0.074 false true
23790 http-nio-8868-exec-67 main 5 WAITING 0.0 0.000 0:0.095 false true
23793 http-nio-8868-exec-68 main 5 WAITING 0.0 0.000 0:0.150 false true
23794 http-nio-8868-exec-69 main 5 WAITING 0.0 0.000 0:0.040 false true
23796 http-nio-8868-exec-70 main 5 WAITING 0.0 0.000 0:0.093 false true
23797 http-nio-8868-exec-71 main 5 WAITING 0.0 0.000 0:0.003 false true
23799 http-nio-8868-exec-72 main 5 WAITING 0.0 0.000 0:0.002 false true
23800 http-nio-8868-exec-73 main 5 WAITING 0.0 0.000 0:0.004 false true
23801 http-nio-8868-exec-74 main 5 WAITING 0.0 0.000 0:0.017 false true
23802 http-nio-8868-exec-75 main 5 WAITING 0.0 0.000 0:0.015 false true
23803 http-nio-8868-exec-76 main 5 WAITING 0.0 0.000 0:0.055 false true
23805 http-nio-8868-exec-77 main 5 WAITING 0.0 0.000 0:0.007 false true
23806 http-nio-8868-exec-78 main 5 WAITING 0.0 0.000 0:0.035 false true
23807 http-nio-8868-exec-79 main 5 WAITING 0.0 0.000 0:0.016 false true
23808 http-nio-8868-exec-80 main 5 WAITING 0.0 0.000 0:0.225 false true
23814 http-nio-8868-exec-81 main 5 WAITING 0.0 0.000 0:0.056 false true
23816 http-nio-8868-exec-82 main 5 WAITING 0.0 0.000 0:0.015 false true
23817 http-nio-8868-exec-83 main 5 WAITING 0.0 0.000 0:0.011 false true
23818 http-nio-8868-exec-84 main 5 WAITING 0.0 0.000 0:0.005 false true
23819 http-nio-8868-exec-85 main 5 WAITING 0.0 0.000 0:0.025 false true
23820 http-nio-8868-exec-86 main 5 WAITING 0.0 0.000 0:0.022 false true
23821 http-nio-8868-exec-87 main 5 WAITING 0.0 0.000 0:0.055 false true
23822 http-nio-8868-exec-88 main 5 WAITING 0.0 0.000 0:0.029 false true
23824 http-nio-8868-exec-89 main 5 WAITING 0.0 0.000 0:0.005 false true
23825 http-nio-8868-exec-90 main 5 WAITING 0.0 0.000 0:0.061 false true
23826 http-nio-8868-exec-91 main 5 WAITING 0.0 0.000 0:0.020 false true
23828 http-nio-8868-exec-92 main 5 WAITING 0.0 0.000 0:0.022 false true
23829 http-nio-8868-exec-93 main 5 WAITING 0.0 0.000 0:0.031 false true
23830 http-nio-8868-exec-94 main 5 WAITING 0.0 0.000 0:0.003 false true
23831 http-nio-8868-exec-95 main 5 WAITING 0.0 0.000 0:0.017 false true
23832 http-nio-8868-exec-96 main 5 WAITING 0.0 0.000 0:0.088 false true
23833 http-nio-8868-exec-97 main 5 WAITING 0.0 0.000 0:0.079 false true
23836 http-nio-8868-exec-98 main 5 WAITING 0.0 0.000 0:0.065 false true
23837 http-nio-8868-exec-99 main 5 WAITING 0.0 0.000 0:0.016 false true
23839 http-nio-8868-exec-100 main 5 WAITING 0.0 0.000 0:0.031 false true
23840 http-nio-8868-exec-101 main 5 WAITING 0.0 0.000 0:0.029 false true
23841 http-nio-8868-exec-102 main 5 WAITING 0.0 0.000 0:0.021 false true
23843 http-nio-8868-exec-103 main 5 WAITING 0.0 0.000 0:0.023 false true
23844 http-nio-8868-exec-104 main 5 WAITING 0.0 0.000 0:0.041 false true
23845 http-nio-8868-exec-105 main 5 WAITING 0.0 0.000 0:0.026 false true
23846 http-nio-8868-exec-106 main 5 WAITING 0.0 0.000 0:0.015 false true
23847 http-nio-8868-exec-107 main 5 WAITING 0.0 0.000 0:0.010 false true
23848 http-nio-8868-exec-108 main 5 WAITING 0.0 0.000 0:0.002 false true
23849 http-nio-8868-exec-109 main 5 WAITING 0.0 0.000 0:0.047 false true
23851 http-nio-8868-exec-110 main 5 WAITING 0.0 0.000 0:0.040 false true
23852 http-nio-8868-exec-111 main 5 WAITING 0.0 0.000 0:0.002 false true
23853 http-nio-8868-exec-112 main 5 WAITING 0.0 0.000 0:0.026 false true
23854 http-nio-8868-exec-113 main 5 WAITING 0.0 0.000 0:0.014 false true
23855 http-nio-8868-exec-114 main 5 WAITING 0.0 0.000 0:0.007 false true
23856 http-nio-8868-exec-115 main 5 WAITING 0.0 0.000 0:0.024 false true
23857 http-nio-8868-exec-116 main 5 WAITING 0.0 0.000 0:0.023 false true
23858 http-nio-8868-exec-117 main 5 WAITING 0.0 0.000 0:0.024 false true
23859 http-nio-8868-exec-118 main 5 WAITING 0.0 0.000 0:0.022 false true
23860 http-nio-8868-exec-119 main 5 WAITING 0.0 0.000 0:0.018 false true
23861 http-nio-8868-exec-120 main 5 WAITING 0.0 0.000 0:0.003 false true
23862 http-nio-8868-exec-121 main 5 WAITING 0.0 0.000 0:0.011 false true
23863 http-nio-8868-exec-122 main 5 WAITING 0.0 0.000 0:0.027 false true
23864 http-nio-8868-exec-123 main 5 WAITING 0.0 0.000 0:0.033 false true
23865 http-nio-8868-exec-124 main 5 WAITING 0.0 0.000 0:0.042 false true
23867 http-nio-8868-exec-125 main 5 WAITING 0.0 0.000 0:0.014 false true
23868 http-nio-8868-exec-126 main 5 WAITING 0.0 0.000 0:0.023 false true
23869 http-nio-8868-exec-127 main 5 WAITING 0.0 0.000 0:0.028 false true
23870 http-nio-8868-exec-128 main 5 WAITING 0.0 0.000 0:0.124 false true
23871 http-nio-8868-exec-129 main 5 WAITING 0.0 0.000 0:0.049 false true
23873 http-nio-8868-exec-130 main 5 WAITING 0.0 0.000 0:0.015 false true
23874 http-nio-8868-exec-131 main 5 WAITING 0.0 0.000 0:0.002 false true
23875 http-nio-8868-exec-132 main 5 WAITING 0.0 0.000 0:0.008 false true
23876 http-nio-8868-exec-133 main 5 WAITING 0.0 0.000 0:0.024 false true
23877 http-nio-8868-exec-134 main 5 WAITING 0.0 0.000 0:0.061 false true
23878 http-nio-8868-exec-135 main 5 WAITING 0.0 0.000 0:0.031 false true
23880 http-nio-8868-exec-136 main 5 WAITING 0.0 0.000 0:0.015 false true
23881 http-nio-8868-exec-137 main 5 WAITING 0.0 0.000 0:0.004 false true
23882 http-nio-8868-exec-138 main 5 WAITING 0.0 0.000 0:0.003 false true
23883 http-nio-8868-exec-139 main 5 WAITING 0.0 0.000 0:0.006 false true
23884 http-nio-8868-exec-140 main 5 WAITING 0.0 0.000 0:0.017 false true
23885 http-nio-8868-exec-141 main 5 WAITING 0.0 0.000 0:0.066 false true
23887 http-nio-8868-exec-142 main 5 WAITING 0.0 0.000 0:0.002 false true
23888 http-nio-8868-exec-143 main 5 WAITING 0.0 0.000 0:0.028 false true
23889 http-nio-8868-exec-144 main 5 WAITING 0.0 0.000 0:0.059 false true
23890 http-nio-8868-exec-145 main 5 WAITING 0.0 0.000 0:0.004 false true
23891 http-nio-8868-exec-146 main 5 WAITING 0.0 0.000 0:0.032 false true
23892 http-nio-8868-exec-147 main 5 WAITING 0.0 0.000 0:0.021 false true
23893 http-nio-8868-exec-148 main 5 WAITING 0.0 0.000 0:0.074 false true
23894 http-nio-8868-exec-149 main 5 WAITING 0.0 0.000 0:0.056 false true
23896 http-nio-8868-exec-150 main 5 WAITING 0.0 0.000 0:0.006 false true
23897 http-nio-8868-exec-151 main 5 WAITING 0.0 0.000 0:0.026 false true
23898 http-nio-8868-exec-152 main 5 WAITING 0.0 0.000 0:0.007 false true
23899 http-nio-8868-exec-153 main 5 WAITING 0.0 0.000 0:0.079 false true
23900 http-nio-8868-exec-154 main 5 WAITING 0.0 0.000 0:0.047 false true
23902 http-nio-8868-exec-155 main 5 WAITING 0.0 0.000 0:0.027 false true
23903 http-nio-8868-exec-156 main 5 WAITING 0.0 0.000 0:0.081 false true
23905 http-nio-8868-exec-157 main 5 WAITING 0.0 0.000 0:0.017 false true
23906 http-nio-8868-exec-158 main 5 WAITING 0.0 0.000 0:0.010 false true
23907 http-nio-8868-exec-159 main 5 WAITING 0.0 0.000 0:0.002 false true
23908 http-nio-8868-exec-160 main 5 WAITING 0.0 0.000 0:0.013 false true
23909 http-nio-8868-exec-161 main 5 WAITING 0.0 0.000 0:0.032 false true
23910 http-nio-8868-exec-162 main 5 WAITING 0.0 0.000 0:0.011 false true
23911 http-nio-8868-exec-163 main 5 WAITING 0.0 0.000 0:0.015 false true
23912 http-nio-8868-exec-164 main 5 WAITING 0.0 0.000 0:0.028 false true
23913 http-nio-8868-exec-165 main 5 WAITING 0.0 0.000 0:0.023 false true
23914 http-nio-8868-exec-166 main 5 WAITING 0.0 0.000 0:0.027 false true
23915 http-nio-8868-exec-167 main 5 WAITING 0.0 0.000 0:0.002 false true
23916 http-nio-8868-exec-168 main 5 WAITING 0.0 0.000 0:0.036 false true
23917 http-nio-8868-exec-169 main 5 WAITING 0.0 0.000 0:0.010 false true
23918 http-nio-8868-exec-170 main 5 WAITING 0.0 0.000 0:0.009 false true
23919 http-nio-8868-exec-171 main 5 WAITING 0.0 0.000 0:0.031 false true
23920 http-nio-8868-exec-172 main 5 WAITING 0.0 0.000 0:0.060 false true
23921 http-nio-8868-exec-173 main 5 WAITING 0.0 0.000 0:0.019 false true
23923 http-nio-8868-exec-174 main 5 WAITING 0.0 0.000 0:0.006 false true
23924 http-nio-8868-exec-175 main 5 WAITING 0.0 0.000 0:0.099 false true
23925 http-nio-8868-exec-176 main 5 WAITING 0.0 0.000 0:0.022 false true
23927 http-nio-8868-exec-177 main 5 WAITING 0.0 0.000 0:0.025 false true
23929 http-nio-8868-exec-178 main 5 WAITING 0.0 0.000 0:0.011 false true
23930 http-nio-8868-exec-179 main 5 WAITING 0.0 0.000 0:0.009 false true
23931 http-nio-8868-exec-180 main 5 WAITING 0.0 0.000 0:0.009 false true
23932 http-nio-8868-exec-181 main 5 WAITING 0.0 0.000 0:0.006 false true
23933 http-nio-8868-exec-182 main 5 WAITING 0.0 0.000 0:0.020 false true
23934 http-nio-8868-exec-183 main 5 WAITING 0.0 0.000 0:0.038 false true
23935 http-nio-8868-exec-184 main 5 WAITING 0.0 0.000 0:0.074 false true
23937 http-nio-8868-exec-185 main 5 WAITING 0.0 0.000 0:0.007 false true
23938 http-nio-8868-exec-186 main 5 WAITING 0.0 0.000 0:0.013 false true
23939 http-nio-8868-exec-187 main 5 WAITING 0.0 0.000 0:0.005 false true
23940 http-nio-8868-exec-188 main 5 WAITING 0.0 0.000 0:0.003 false true
23941 http-nio-8868-exec-189 main 5 WAITING 0.0 0.000 0:0.008 false true
23942 http-nio-8868-exec-190 main 5 WAITING 0.0 0.000 0:0.013 false true
23943 http-nio-8868-exec-191 main 5 WAITING 0.0 0.000 0:0.004 false true
23944 http-nio-8868-exec-192 main 5 WAITING 0.0 0.000 0:0.007 false true
23945 http-nio-8868-exec-193 main 5 WAITING 0.0 0.000 0:0.011 false true
23946 http-nio-8868-exec-194 main 5 WAITING 0.0 0.000 0:0.032 false true
23947 http-nio-8868-exec-195 main 5 WAITING 0.0 0.000 0:0.024 false true
23948 http-nio-8868-exec-196 main 5 WAITING 0.0 0.000 0:0.031 false true
23949 http-nio-8868-exec-197 main 5 WAITING 0.0 0.000 0:0.037 false true
23950 http-nio-8868-exec-198 main 5 WAITING 0.0 0.000 0:0.065 false true
23951 http-nio-8868-exec-199 main 5 WAITING 0.0 0.000 0:0.009 false true
23953 http-nio-8868-exec-200 main 5 WAITING 0.0 0.000 0:0.043 false true通过随机挑选了几个线程打印此时运行堆栈信息发现他们都在等待 product-sync-source-data-service2 这个线程释放锁。
然后查看 product-sync-source-data-service2 这个线程发现他一直阻塞在打印日志这一步，对应 JAVA 代码如上图右一。
[arthas@1]$ thread 639
"product-sync-source-data-service2" Id=639 RUNNABLE (in native)
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:326)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:122)
at java.io.PrintStream.write(PrintStream.java:480)
at java.io.FilterOutputStream.write(FilterOutputStream.java:97)
at ch.qos.logback.core.joran.spi.ConsoleTarget$1.write(ConsoleTarget.java:37)
at ch.qos.logback.core.OutputStreamAppender.writeBytes(OutputStreamAppender.java:199)
at ch.qos.logback.core.OutputStreamAppender.subAppend(OutputStreamAppender.java:231)
at ch.qos.logback.core.OutputStreamAppender.append(OutputStreamAppender.java:102)
at ch.qos.logback.core.UnsynchronizedAppenderBase.doAppend(UnsynchronizedAppenderBase.java:84)
at ch.qos.logback.core.spi.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:51)
at ch.qos.logback.classic.Logger.appendLoopOnAppenders(Logger.java:270)
at ch.qos.logback.classic.Logger.callAppenders(Logger.java:257)
at ch.qos.logback.classic.Logger.buildLoggingEventAndAppend(Logger.java:421)
at ch.qos.logback.classic.Logger.filterAndLog_0_Or3Plus(Logger.java:383)
at ch.qos.logback.classic.Logger.info(Logger.java:591)
at com.ly.base.log.Logger.info(Logger.java:74)
at com.ly.mc.ticket.service.product.item.ProductSyncDataService.lambda$syncProductItemDetails$232(ProductSyncDataService.java:130)
at com.ly.mc.ticket.service.product.item.ProductSyncDataService$$Lambda$1687/1823423302.run(Unknown Source)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)一开始一直以为是写入本地文件阻塞，没仔细看堆栈信息。通过网络搜索 RUNNABLE (in native) at java.io.FileOutputStream.writeBytes(Native Method) 发现此类问题出现的还挺多，可以确认是因为日志会输出到控制台中，而 Docker 容器的控制台在日志过长或是很多的时候会 hangs
- https://segmentfault.com/a/1190000018240611
- https://blog.csdn.net/qq_26394851/article/details/114123000
- https://github.com/moby/moby/issues/35865
之后通过翻阅源码可以看到：
- 我们使用的
com.ly.base.log.Logger内部通过静态变量持有了一个 org.slf4j.Logger 对象实例 org.slf4j.Logger最终会调用OutputStreamAppender对象的writeBytes方法写入日志，此时会加锁- 由于开启了日志控制台输出，所以最终走到了
System.out.write输出到了容器的控制台中
四、优化手段
- 生产环境屏蔽控制台的日志输出
- 从最后的根因来看，驻留的很大原因是爬虫线程死锁导致相关线程内的资源未被释放。↩