融合Swin Transformer的虫害图像实例分割优化方法研究

doi:10.12302/j.issn.1000-2006.202206048

国家林草科技领军期刊
中国精品科技期刊
中国高校百佳科技期刊
江苏省新闻出版政府奖期刊奖
RCCSE林学权威期刊（A+）
CSCD核心期刊
Scopus数据库收录期刊
中文核心期刊
SCD核心期刊

作者加群：102861116

微信公众号：南京林业大学学报

高级检索

南京林业大学学报（自然科学版） ›› 2023, Vol. 47 ›› Issue (3): 1-10.doi: 10.12302/j.issn.1000-2006.202206048

所属专题：第三届中国林草计算机应用大会论文精选

• 专题报道：第三届中国林草计算机应用大会论文精选（执行主编李凤日） • 上一篇下一篇

融合Swin Transformer的虫害图像实例分割优化方法研究

高家军(), 张旭, 郭颖(), 刘昱坤, 郭安琪, 石蒙蒙, 王鹏, 袁莹

中国林业科学研究院资源信息研究所,北京 100091

收稿日期:2022-06-25 修回日期:2022-12-27 出版日期:2023-05-30 发布日期:2023-05-25
通讯作者: 郭颖
基金资助:
中央级公益性科研院所基本科研业务费专项资金项目(CAFYBB2021ZB002)

Research on the optimized pest image instance segmentation method based on the Swin Transformer model

GAO Jiajun(), ZHANG Xu, GUO Ying(), LIU Yukun, GUO Anqi, SHI Mengmeng, WANG Peng, YUAN Ying

Institute of Forest Resource Information Techniques, Chinese Academy of Forestry, Beijing 100091, China

Received:2022-06-25 Revised:2022-12-27 Online:2023-05-30 Published:2023-05-25
Contact: GUO Ying

摘要/Abstract

摘要：

【目的】为了实现对虫害的精准监测,提出了一种融合Swin Transformer的图像实例分割优化方法,以期有效解决复杂真实场景下多幼虫个体图像识别分割困难的问题。【方法】选用Swin Transformer模型,改进Mask R-CNN实例分割模型的主干网部分,对黄野螟幼虫虫害图像进行识别分割。针对不同结构参数的Swin Transformer模型与ResNet模型,调整各层的输入输出维度,将其分别设置为Mask R-CNN的主干网进行对比实验,从定量与定性两个角度分析不同主干网的Mask R-CNN模型对黄野螟幼虫的识别分割精度与效果,确定最佳模型结构。【结果】①该方法在虫害识别框选方面的测度(F1)分数可达89.7%,平均精度(A_P)可达88.0%;在虫害识别分割方面的F1分数可达84.3%,A_P可达82.2%。相较于Mask R-CNN,在目标框选与目标分割方面分别提升8.75%与8.40%。②对于小目标虫害识别分割任务,该方法在虫害识别框选方面的F1分数可达88.4%,A_P可达86.3%;在虫害识别分割方面的F1分数可达84.0%,A_P可达81.7%。相较于Mask R-CNN,在目标框选与目标分割方面分别提升9.30%与9.45%。【结论】对于复杂真实场景下的图像实例分割任务,其识别分割效果极大地依赖于模型对图像特征的提取能力,而融合了Swin Transformer的Mask R-CNN实例分割模型,在主干网的特征提取能力更强,模型整体的识别分割效果更好,可为虫害的识别监测提供技术支撑,同时为保护农、林、牧等产业资源提供解决方案。

关键词: 虫害识别, Swin Transformer, Mask R-CNN, 实例分割, 土沉香, 黄野螟

Abstract:

【Objective】To achieve accurate pest monitoring, the author proposes an optimized instance segmentation method based on the Swin Transformer to effectively solve the difficulty in image recognition and segmentation of multi-larval individuals under complex real scenarios.【Method】The Swin Transformer model was selected to improve the backbone network of the Mask R-CNN instance segmentation model and to identify and segment Heortia vitessoides larvae which harmed Aquilaria sinensis. The input and output dimensions of all layers of the Swin Transformer and ResNet models with different structural parameters were adjusted. Both models were set as the backbone networks of Mask R-CNN for comparative experiments. H. vitessoides moore larvae identification and segmentation performances for different backbone networks were quantitatively and qualitatively analyzed using Mask R-CNN models to determine the best model structure.【Result】(1) Using this method, the F1 score and A_P were 89.7% and 88.0%, respectively, in terms of pest identification framing, and 84.3% and 82.2%, respectively, in terms of pest identification and segmentation, increasing by 8.75% and 8.40%, respectively, compared to that of the Mask R-CNN model in terms of target framing and segmentation. (2) For small target pest identification and segmentation tasks, the F1 score and A_P were 88.4% and 86.3%, respectively, in terms of pest identification framing, 84.0% and 81.7%, respectively, in terms of pest identification and segmentation, and increased by 9.30% and 9.45%, respectively, compared to that of the Mask R-CNN model in terms of target framing and segmentation.【Conclusion】In segmentation tasks under complex real scenarios, the recognition and segmentation effects depend to a large extent on the model’s ability to extract image features. By integrating the Swin Transformer, the mask R-CNN instance segmentation model has a stronger ability to extract features in the backbone network, with a better overall recognition and segmentation effect. It could provide technical support for the identification and monitoring of pests and solutions for the protection of agriculture, forestry, animal husbandry, and other industrial resources.

Key words: pest recognition, Swin Transformer, Mask R-CNN, instance segmentation, Aguilaria sinensis, Heortia vitessoides

中图分类号:

高家军,张旭,郭颖,等. 融合Swin Transformer的虫害图像实例分割优化方法研究[J]. 南京林业大学学报（自然科学版）, 2023, 47(3): 1-10.

GAO Jiajun, ZHANG Xu, GUO Ying, LIU Yukun, GUO Anqi, SHI Mengmeng, WANG Peng, YUAN Ying. Research on the optimized pest image instance segmentation method based on the Swin Transformer model[J].Journal of Nanjing Forestry University (Natural Science Edition), 2023, 47(3): 1-10.DOI: 10.12302/j.issn.1000-2006.202206048.

图/表 12

图1

图2

图3

图4

图5

图6

图7

图8

表1

图9

表2

图10

参考文献 37

[1]	徐锡祥. 林业病虫害特点、原因及综合防治解析[J]. 新农业, 2021(22):22.
	XU X X. Analysis on characteristics,causes and integrated control of forest diseases and pests[J]. Modern Agriculture, 2021(22):22.
[2]	张旭. 森林病虫害的发生特点及综合防治技术[J]. 农业灾害研究, 2021, 11(7):21-22.
	ZHANG X. Occurrence characteristics and integrated control technology of forest diseases and insect pests[J]. J Agric Catastrophology, 2021, 11(7):21-22.
[3]	樊巍, 苑静, 赵波. 森林病虫害的发生特点及综合防治技术[J]. 农业与技术, 2019, 39(23):70-71.
	FAN W, YUAN J, ZHAO B. Occurrence characteristics and integrated control techniques of forest diseases and insect pests[J]. Agric Technol, 2019, 39(23):70-71.DOI:10.19754/j.nyyjs.20191215028.
[4]	NASSER SHAH N A, OSMAN M K, OTHMAN N A, et al. Identification and counting of brown planthopper in paddy field using image processing techniques[J]. Procedia Comput Sci, 2019, 163:580-590.DOI:10.1016/j.procs.2019.12.140.
[5]	WEN C L, WU D X, HU H S, et al. Pose estimation-dependent identification method for field moth images using deep learning architecture[J]. Biosyst Eng, 2015, 136:117-128.DOI:10.1016/j.biosystemseng.2015.06.002.
[6]	WANG R J, ZHANG J, DONG W, et al. A crop pests image classification algorithm based on deep convolutional neural network[J]. TELKOMNIKA (Telecommun Comput Electron Control), 2017, 15(3):1239.DOI:10.12928/TELKOMNIKA.v15i3.5382.
[7]	李静, 陈桂芬, 安宇. 基于优化卷积神经网络的玉米螟虫害图像识别[J]. 华南农业大学学报, 2020, 41(3):110-116.
	LI J, CHEN G F, AN Y. Image recognition of Pyrausta nubilalis based on optimized convolutional neural network[J]. J South China Agric Univ, 2020, 41(3):110-116.DOI:10.7671/j.issn.1001-411X.201907017.
[8]	KARAR M E, ALSUNAYDI F, ALBUSAYMI S, et al. A new mobile application of agricultural pests recognition using deep learning in cloud computing system[J]. Alex Eng J, 2021, 60(5):4423-4432.DOI:10.1016/j.aej.2021.03.009.
[9]	HONG S J, KIM S Y, KIM E, et al. Moth detection from pheromone trap images using deep learning object detectors[J]. Agriculture, 2020, 10(5):170.DOI:10.3390/agriculture10050170.
[10]	GONÇALVES J P, PINTO F A C, QUEIROZ D M, et al. Deep learning architectures for semantic segmentation and automatic estimation of severity of foliar symptoms caused by diseases or pests[J]. Biosyst Eng, 2021, 210:129-142.DOI:10.1016/j.biosystemseng.2021.08.011.
[11]	田洪宝. 基于深度卷积神经网络的林区航拍图像虫害区域分割[D]. 北京: 北京林业大学, 2019.
	TIAN H B. Pest region segmentation of aerial photography image in forest region based on deep convolution neural networks[D]. Beijing: Beijing Forestry University, 2019.
[12]	陈冬梅, 张赫, 魏凯华, 等. 复杂背景下昆虫图像的快速分割与识别[J]. 江苏农业科学, 2021, 49(24):195-204.
	CHEN D M, ZHANG H, WEI K H, et al. Fast segmentation and recognition of insect images in complex background[J]. Jiangsu Agric Sci, 2021, 49(24):195-204. DOI: 10.15889/j.issn.1002-1302.2021.24.034.
[13]	王卫民, 符首夫, 顾榕蓉, 等. 基于卷积神经网络的虫情图像分割和计数方法[J]. 计算机工程与科学, 2020, 42(1):110-116.
	WANG W M, FU S F, GU R R, et al. An insect image segmentation and counting method based on convolutional neural network[J]. Comput Eng & Sci, 2020, 42(1):110-116.DOI:10.3969/j.issn.1007-130X.2020.01.0114.
[14]	张善文, 邵彧, 齐国红, 等. 基于多尺度注意力卷积网络的作物害虫检测[J]. 江苏农业学报, 2021, 37(3):579-588.
	ZHANG S W, SHAO Y, QI G H, et al. Crop pest detection based on multi-scale convolutional network with attention[J]. Jiangsu J Agr Sci, 2021, 37(3):579-588.DOI:10.3969/j.issn.1000-4440.2021.03.005.
[15]	谭富祥, 钱育蓉, 孔钰婷, 等. 基于Transformer的多分支单图像去雨方法[J]. 计算机应用研究, 2022, 39(8):2500-2505,2519.
	TAN F X, QIAN Y R, KONG Y T, et al. Multi-branch single image deraining network based on Transformer[J]. Appl Res Comput, 2022, 39(8):2500-2505, 2519.DOI:10.19734/j.issn.1001-3695.2021.12.0695.
[16]	陈敏, 王君, 董明利, 等. 改进的Mask R-CNN多尺度实例分割算法研究[J]. 激光杂志, 2020, 41(5):40-44.
	CHEN M, WANG J, DONG M L, et al. Research on improved Mask R-CNN for multi-scale instance segmentation method[J]. Laser J, 2020, 41(5):40-44.DOI:10.14016/j.cnki.jgzz.2020.05.040.
[17]	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all You need[EB/OL].[2022-12-01]. http://arxiv.org/abs/1706.03762.
[18]	田应仲, 卜雪虎. 基于注意力机制与Swin Transformer模型的腰椎图像分割方法[J]. 计量与测试技术, 2021, 48(12):57-61.
	TIAN Y Z, BU X H. Lumbar spine image segmentation method based on attention mechanism and Swin Transformer model[J]. Metrol & Meas Tech, 2021, 48(12):57-61.DOI:10.15988/j.cnki.1004-6941.2021.12.013.
[19]	张重生, 陈杰, 纵瑞星, 等. 基于Transformer的低质场景字符检测算法[J]. 北京邮电大学学报, 2022, 45(2):124-130.
	ZHANG C S, CHEN J, ZONG R X, et al. Transformer based scene character detection over low quality images[J]. J Beijing Univ Posts Telecommun, 2022, 45(2):124-130.DOI:10.13190/j.jbupt.2021-155.
[20]	LIU Z, LIN Y T, CAO Y, et al. Swin transformer:hierarchical vision transformer using shifted windows[C]// IEEE/CVF International Conference on Computer Vision (ICCV).Montreal,QC,Canada.IEEE, 2021:9992-10002.DOI:10.1109/ICCV48922.2021.00986.
[21]	HE K M, GKIOXARI G, DOLLÁR P, et al. Mask R-CNN[C]// IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas,NV,USA: IEEE, 2017:2961-2969.DOI:10.1109/ICCV.2017.322.
[22]	蒋开彬, 祝文娟, 潘文, 等. 基于SRAP标记的土沉香遗传多样性分析[J]. 中南林业科技大学学报, 2020, 40(1):131-136.
	JIANG K B, ZHU W J, PAN W, et al. Analysis of genetic diversity of Aquilaria sinensis(Lour.) Gilg based on SRAP markers[J]. J Central South Univ For & Technol, 2020, 40(1):131-136.DOI:10.14067/j.cnki.1673-923x.2020.01.016.
[23]	庞圣江, 张培, 杨保国, 等. 林隙大小对土沉香人工更新幼树生长发育的影响[J]. 西北农林科技大学学报(自然科学版), 2020, 48(4):83-88.
	PANG S J, ZHANG P, YANG B G, et al. Effects of gap size on growth of transplanted saplings of Aquilaria sinensis[J]. J Northwest A & F Univ (Nat Sci Ed), 2020, 48(4):83-88.DOI:10.13207/j.cnki.jnwafu.2020.04.011.
[24]	张小霞. 土沉香开发利用研究进展[J]. 防护林科技, 2020(4):63-66.
	ZHANG X X. Research progress on development and utilization of Aquilaria sinensis[J]. Prot For Sci Technol, 2020(4):63-66.DOI:10.13601/j.issn.1005-5215.2020.04.024.
[25]	宋晓琛, 王西洋, 杨光, 等. 无机盐与激素混合对土沉香结香的诱导[J]. 林业科学, 2020, 56(8):121-130.
	SONG X C, WANG X Y, YANG G, et al. Mechanism of agarwood formation under the induction of both inorganic salts and hormones[J]. Sci Silvae Sin, 2020, 56(8):121-130.DOI:10.11707/j.1001-7488.20200814.
[26]	洪仁辉, 尹吉锋, 陈彧, 等. 白木香重要害虫黄野螟研究进展[J]. 热带林业, 2019, 47(3):66-68.
	HONG R H, YIN J F, CHEN Y, et al. Advances in research on the Heortia vitessoides Moore on Aquilaria sinensis[J]. Trop For, 2019, 47(3):66-68.DOI:10.3969/j.issn.1672-0938.2019.03.015.
[27]	王忠, 谢伟忠, 朱诚棋, 等. 黄野螟的羽化和生殖行为节律[J]. 中国森林病虫, 2018, 37(1):24-27,30.
	WANG Z, XIE W Z, ZHU C Q, et al. Circadian rhythm of emergence and reproduction of Heortia vitessoides Moore (Lepidoptera:Crambidae)[J]. For Pest Dis, 2018, 37(1):24-27,30.
[28]	茅裕婷, 张蒙, 靳秀芳, 等. 土沉香对黄野螟的抗性研究[J]. 华南农业大学学报, 2017, 38(6):89-96.
	MAO Y T, ZHANG M, JIN X F, et al. Study on resistance of Aquilaria sinensis against Heortia vitessoides[J]. J South China Agric Univ, 2017, 38(6):89-96.DOI:10.7671/j.issn.1001-411X.2017.06.014.
[29]	严珍, 岳建军. 温度及补充营养对黄野螟生长发育和繁殖的影响[J]. 热带作物学报, 2019, 40(9):1789-1795.
	YAN Z, YUE J J. Effects of temperature and supplementary foods on the development and fecundity of Heortia vitessoides[J]. Chin J Trop Crops, 2019, 40(9):1789-1795.DOI:10.3969/j.issn.1000-2561.2019.09.017.
[30]	高新波, 莫梦竟成, 汪海涛, 等. 小目标检测研究进展[J]. 数据采集与处理, 2021, 36(3):391-417.
	GAO X B, MO M J C, WANG H T, et al. Recent advances in small object detection[J]. J Data Acquis Process, 2021, 36(3):391-417.DOI:10.16337/j.1004-9037.2021.03.001.
[31]	LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]//IEEE Conference on Computer Vision and Pattern Recognition.Boston,MA, USA:IEEE, 2015:3431-3440.DOI:10.1109/CVPR.2015.7298965.
[32]	HINTON G E, SRIVASTAVA N, KRIZHEVSKY A, et al. Improving neural networks by preventing co-adaptation of feature detectors[EB/OL].[2022-12-01]. http://arxiv.org/abs/1207.0580.
[33]	张海燕, 徐心语, 马雪芬, 等. 超声图像中复合材料褶皱形态的Mask-RCNN识别方法[J]. 物理学报, 2022, 71(7):074302.
	ZHANG H Y, XU X Y, MA X F, et al. Mask-RCNN recognition method of composite fold shape in ultrasound images[J]. Acta Phys Sin, 2022, 71(7):074302.DOI:10.7498/aps.71.20212009.
[34]	LIN T L, CHANG H Y, CHEN K H. Pest and disease identification in the growth of sweet peppers using faster R-CNN[J]. IEEE Int Conf Consumer Electron Taiwan (ICCE TW), 2019:1-2.DOI:10.1109/ICCE-TW46550.2019.8991893.
[35]	周维, 牛永真, 王亚炜, 等. 基于改进的YOLOv4-GhostNet水稻病虫害识别方法[J]. 江苏农业学报, 2022, 38(3):685-695.
	ZHOU W, NIU Y Z, WANG Y W, et al. Rice pests and diseases identification method based on improved YOLOv4-GhostNet[J]. Jiangsu J Agr Sci, 2022, 38(3):685-695.DOI:10.3969/j.issn.1000-4440.2022.03.014.
[36]	刘晨曦, 刘大铭, 杨芳, 等. 基于改进水平集的水稻虫害分割算法[J]. 宁夏大学学报(自然科学版), 2019, 40(3):246-254.
	LIU C X, LIU D M, YANG F, et al. Partitioning algorithm for rice pest based on improved level set method[J]. J Ningxia Univ (Nat Sci Ed), 2019, 40(3):246-254.DOI:10.3969/j.issn.0253-2328.2019.03.010.
[37]	卜俊怡, 孙国祥, 王迎旭, 等. 基于诱虫板图像的温室番茄作物害虫识别与监测方法[J]. 南京农业大学学报, 2021, 44(2):373-383.
	BU J Y, SUN G X, WANG Y X, et al. Identification and monitoring method of tomato crop pests in greenhouse based on trapping board image[J]. J Nanjing Agric Univ, 2021, 44(2):373-383.DOI:10.7685/jnau.202005031.

模型 model	F1分数/% F1 score	平均精度/% A_P	小目标 F1分数/% F1 score of small target	小目标平均精度/% A_P of small target	参数大小/MB parameter size
PST(box)	89.7	88.0	88.4	86.3	180
PST(seg)	84.3	82.2	84.0	81.7
PST-S(box)	88.9	87.1	88.3	86.6	262
PST-S(seg)	83.8	82.2	82.8	81.1
MRC 50(box)	80.3	78.6	78.3	76.6	168
MRC 50(seg)	75.7	73.8	74.6	72.8
MRC 101(box)	81.7	79.5	79.9	77.3	240
MRC 101(seg)	76.0	73.5	74.5	72.0

模型 models	计算量/ GFLOPs computation	参数量/ ×10⁶ parameters	检测速度/ (张·s^-1) picture	误检量/个 false detections	漏检量/个 missed detections
PST	135.38	47.37	2.5	14	30
PST-S	287.45	68.69	2.3	17	41
MRC 50	329.33	43.75	3.4	9	66
MRC 101	481.48	62.74	2.6	22	43

融合Swin Transformer的虫害图像实例分割优化方法研究

Research on the optimized pest image instance segmentation method based on the Swin Transformer model

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 12

参考文献 37

相关文章 15

编辑推荐

Metrics

本文评价

作者

读者

审稿

[1]	向俊, 严恩萍, 姜镓伟, 宋亚斌, 韦维, 莫登奎. 基于全卷积神经网络和低分辨率标签的森林变化检测研究[J]. 南京林业大学学报（自然科学版）, 2024, 48(1): 187-195.
[2]	李双娴, 陆鑫, 多杰才仁, 张怀清, 薛联凤, 云挺. 一种新的地面激光点云中树木叶面积计算方法[J]. 南京林业大学学报（自然科学版）, 2023, 47(5): 28-38.
[3]	袁莹, 王雪峰, 王甜, 陈飞飞, 黄川腾, 林玲, 董晓娜. 基于多图像特征的幼龄沉香全氮估测[J]. 南京林业大学学报（自然科学版）, 2023, 47(3): 19-28.
[4]	尹显明, 棘玉, 张日清, 莫登奎, 彭邵锋, 韦维. 深度学习在基于叶片的油茶品种识别中的研究[J]. 南京林业大学学报（自然科学版）, 2023, 47(3): 29-36.
[5]	棘玉, 尹显明, 严恩萍, 蒋佳敏, 彭邵锋, 莫登奎. 基于相机拍照的油茶果形状特征提取研究[J]. 南京林业大学学报（自然科学版）, 2022, 46(2): 63-70.
[6]	姜金岑, 李联队, 王美丽. 基于L1中值骨架提取的植物茎干补全研究[J]. 南京林业大学学报（自然科学版）, 2022, 46(1): 40-50.
[7]	王羽尘, 马健霄, 刘宇航, 白莹佳. 基于烟气扩散特征的林区隧道火灾人群疏散模型[J]. 南京林业大学学报（自然科学版）, 2021, 45(6): 177-184.
[8]	刘嘉政, 王雪峰, 王甜. 基于深度学习的树种图像自动识别[J]. 南京林业大学学报（自然科学版）, 2020, 44(1): 138-144.
[9]	黄笑,云挺,薛联凤,胡春华,陈帮乾. 基于流体运动仿真的不同林冠形状抗风强度分析[J]. 南京林业大学学报（自然科学版）, 2019, 43(02): 107-113.
[10]	杨婷婷,管昉立,徐爱俊. 基于Graph Cut算法的多株立木轮廓提取方法[J]. 南京林业大学学报（自然科学版）, 2018, 42(06): 91-98.
[11]	金大勇. 网络安全体系中蜜罐技术分析[J]. 南京林业大学学报（自然科学版）, 2006, 30(03): 134-135.
[12]	张冬,云挺,薛联凤,阮宏华. 基于力场的点云树木骨架提取方法[J]. 南京林业大学学报（自然科学版）, 2016, 40(02): 160-166.
[13]	胡春华,李萍萍. 树木三维可视化建模技术研究述评[J]. 南京林业大学学报（自然科学版）, 2015, 39(06): 148-154.
[14]	张天安,云挺,薛联凤,安锋. 基于地面激光雷达的活立木枝干三维建模[J]. 南京林业大学学报（自然科学版）, 2015, 39(04): 163-167.
[15]	蒋小川. 节能型无线传感器网络在森林环境监测中的应用[J]. 南京林业大学学报（自然科学版）, 2014, 38(04): 14-18.