Research on the optimized pest image instance segmentation method based on the Swin Transformer model

GAO Jiajun, ZHANG Xu, GUO Ying, LIU Yukun, GUO Anqi, SHI Mengmeng, WANG Peng, YUAN Ying

JOURNAL OF NANJING FORESTRY UNIVERSITY ›› 2023, Vol. 47 ›› Issue (3) : 1-10.

PDF(3804 KB)
南京林业大学学报（自然科学版）
PDF(3804 KB)
JOURNAL OF NANJING FORESTRY UNIVERSITY ›› 2023, Vol. 47 ›› Issue (3) : 1-10. DOI: 10.12302/j.issn.1000-2006.202206048

Research on the optimized pest image instance segmentation method based on the Swin Transformer model

Author information +
History +

Abstract

【Objective】To achieve accurate pest monitoring, the author proposes an optimized instance segmentation method based on the Swin Transformer to effectively solve the difficulty in image recognition and segmentation of multi-larval individuals under complex real scenarios.【Method】The Swin Transformer model was selected to improve the backbone network of the Mask R-CNN instance segmentation model and to identify and segment Heortia vitessoides larvae which harmed Aquilaria sinensis. The input and output dimensions of all layers of the Swin Transformer and ResNet models with different structural parameters were adjusted. Both models were set as the backbone networks of Mask R-CNN for comparative experiments. H. vitessoides moore larvae identification and segmentation performances for different backbone networks were quantitatively and qualitatively analyzed using Mask R-CNN models to determine the best model structure.【Result】(1) Using this method, the F1 score and AP were 89.7% and 88.0%, respectively, in terms of pest identification framing, and 84.3% and 82.2%, respectively, in terms of pest identification and segmentation, increasing by 8.75% and 8.40%, respectively, compared to that of the Mask R-CNN model in terms of target framing and segmentation. (2) For small target pest identification and segmentation tasks, the F1 score and AP were 88.4% and 86.3%, respectively, in terms of pest identification framing, 84.0% and 81.7%, respectively, in terms of pest identification and segmentation, and increased by 9.30% and 9.45%, respectively, compared to that of the Mask R-CNN model in terms of target framing and segmentation.【Conclusion】In segmentation tasks under complex real scenarios, the recognition and segmentation effects depend to a large extent on the model’s ability to extract image features. By integrating the Swin Transformer, the mask R-CNN instance segmentation model has a stronger ability to extract features in the backbone network, with a better overall recognition and segmentation effect. It could provide technical support for the identification and monitoring of pests and solutions for the protection of agriculture, forestry, animal husbandry, and other industrial resources.

Key words

pest recognition / Swin Transformer / Mask R-CNN / instance segmentation / Aguilaria sinensis / Heortia vitessoides

Cite this article

EndNote

Ris (Procite)

Bibtex

Download Citations
GAO Jiajun , ZHANG Xu , GUO Ying , et al . Research on the optimized pest image instance segmentation method based on the Swin Transformer model[J]. JOURNAL OF NANJING FORESTRY UNIVERSITY. 2023, 47(3): 1-10 https://doi.org/10.12302/j.issn.1000-2006.202206048

References

List( Publishing order | Descend order by publishing year | Descend order by cited within ) Chart analysis
[1]
徐锡祥. 林业病虫害特点、原因及综合防治解析[J]. 新农业, 2021(22):22.
XU X X. Analysis on characteristics,causes and integrated control of forest diseases and pests[J]. Modern Agriculture, 2021(22):22.
Cited in this article [1]
[2]
张旭. 森林病虫害的发生特点及综合防治技术[J]. 农业灾害研究, 2021, 11(7):21-22.
ZHANG X. Occurrence characteristics and integrated control technology of forest diseases and insect pests[J]. J Agric Catastrophology, 2021, 11(7):21-22.
Cited in this article [1]
[3]
樊巍, 苑静, 赵波. 森林病虫害的发生特点及综合防治技术[J]. 农业与技术, 2019, 39(23):70-71.
FAN W, YUAN J, ZHAO B. Occurrence characteristics and integrated control techniques of forest diseases and insect pests[J]. Agric Technol, 2019, 39(23):70-71.DOI:10.19754/j.nyyjs.20191215028.
Cited in this article [1]
[4]
NASSER SHAH N A, OSMAN M K, OTHMAN N A, et al. Identification and counting of brown planthopper in paddy field using image processing techniques[J]. Procedia Comput Sci, 2019, 163:580-590.DOI:10.1016/j.procs.2019.12.140.
Cited in this article [1]
[5]
WEN C L, WU D X, HU H S, et al. Pose estimation-dependent identification method for field moth images using deep learning architecture[J]. Biosyst Eng, 2015, 136:117-128.DOI:10.1016/j.biosystemseng.2015.06.002.
Cited in this article [1]
[6]
WANG R J, ZHANG J, DONG W, et al. A crop pests image classification algorithm based on deep convolutional neural network[J]. TELKOMNIKA (Telecommun Comput Electron Control), 2017, 15(3):1239.DOI:10.12928/TELKOMNIKA.v15i3.5382.
Cited in this article [1]
[7]
李静, 陈桂芬, 安宇. 基于优化卷积神经网络的玉米螟虫害图像识别[J]. 华南农业大学学报, 2020, 41(3):110-116.
LI J, CHEN G F, AN Y. Image recognition of Pyrausta nubilalis based on optimized convolutional neural network[J]. J South China Agric Univ, 2020, 41(3):110-116.DOI:10.7671/j.issn.1001-411X.201907017.
Cited in this article [1]
[8]
KARAR M E, ALSUNAYDI F, ALBUSAYMI S, et al. A new mobile application of agricultural pests recognition using deep learning in cloud computing system[J]. Alex Eng J, 2021, 60(5):4423-4432.DOI:10.1016/j.aej.2021.03.009.
Cited in this article [1]
[9]
HONG S J, KIM S Y, KIM E, et al. Moth detection from pheromone trap images using deep learning object detectors[J]. Agriculture, 2020, 10(5):170.DOI:10.3390/agriculture10050170.
Cited in this article [1]
[10]
GONÇALVES J P, PINTO F A C, QUEIROZ D M, et al. Deep learning architectures for semantic segmentation and automatic estimation of severity of foliar symptoms caused by diseases or pests[J]. Biosyst Eng, 2021, 210:129-142.DOI:10.1016/j.biosystemseng.2021.08.011.
Cited in this article [1]
[11]
田洪宝. 基于深度卷积神经网络的林区航拍图像虫害区域分割[D]. 北京: 北京林业大学, 2019.
TIAN H B. Pest region segmentation of aerial photography image in forest region based on deep convolution neural networks[D]. Beijing: Beijing Forestry University, 2019.
Cited in this article [1]
[12]
陈冬梅, 张赫, 魏凯华, 等. 复杂背景下昆虫图像的快速分割与识别[J]. 江苏农业科学, 2021, 49(24):195-204.
CHEN D M, ZHANG H, WEI K H, et al. Fast segmentation and recognition of insect images in complex background[J]. Jiangsu Agric Sci, 2021, 49(24):195-204. DOI: 10.15889/j.issn.1002-1302.2021.24.034.
Cited in this article [1]
[13]
王卫民, 符首夫, 顾榕蓉, 等. 基于卷积神经网络的虫情图像分割和计数方法[J]. 计算机工程与科学, 2020, 42(1):110-116.
WANG W M, FU S F, GU R R, et al. An insect image segmentation and counting method based on convolutional neural network[J]. Comput Eng & Sci, 2020, 42(1):110-116.DOI:10.3969/j.issn.1007-130X.2020.01.0114.
Cited in this article [2]
[14]
张善文, 邵彧, 齐国红, 等. 基于多尺度注意力卷积网络的作物害虫检测[J]. 江苏农业学报, 2021, 37(3):579-588.
ZHANG S W, SHAO Y, QI G H, et al. Crop pest detection based on multi-scale convolutional network with attention[J]. Jiangsu J Agr Sci, 2021, 37(3):579-588.DOI:10.3969/j.issn.1000-4440.2021.03.005.
Cited in this article [1]
[15]
谭富祥, 钱育蓉, 孔钰婷, 等. 基于Transformer的多分支单图像去雨方法[J]. 计算机应用研究, 2022, 39(8):2500-2505,2519.
TAN F X, QIAN Y R, KONG Y T, et al. Multi-branch single image deraining network based on Transformer[J]. Appl Res Comput, 2022, 39(8):2500-2505, 2519.DOI:10.19734/j.issn.1001-3695.2021.12.0695.
Cited in this article [2]
[16]
陈敏, 王君, 董明利, 等. 改进的Mask R-CNN多尺度实例分割算法研究[J]. 激光杂志, 2020, 41(5):40-44.
CHEN M, WANG J, DONG M L, et al. Research on improved Mask R-CNN for multi-scale instance segmentation method[J]. Laser J, 2020, 41(5):40-44.DOI:10.14016/j.cnki.jgzz.2020.05.040.
Cited in this article [1]
[17]
VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all You need[EB/OL].[2022-12-01]. http://arxiv.org/abs/1706.03762.
http://arxiv.org/abs/1706.03762
Cited in this article [1]
[18]
田应仲, 卜雪虎. 基于注意力机制与Swin Transformer模型的腰椎图像分割方法[J]. 计量与测试技术, 2021, 48(12):57-61.
TIAN Y Z, BU X H. Lumbar spine image segmentation method based on attention mechanism and Swin Transformer model[J]. Metrol & Meas Tech, 2021, 48(12):57-61.DOI:10.15988/j.cnki.1004-6941.2021.12.013.
Cited in this article [1]
[19]
张重生, 陈杰, 纵瑞星, 等. 基于Transformer的低质场景字符检测算法[J]. 北京邮电大学学报, 2022, 45(2):124-130.
ZHANG C S, CHEN J, ZONG R X, et al. Transformer based scene character detection over low quality images[J]. J Beijing Univ Posts Telecommun, 2022, 45(2):124-130.DOI:10.13190/j.jbupt.2021-155.
Cited in this article [1]
[20]
LIU Z, LIN Y T, CAO Y, et al. Swin transformer:hierarchical vision transformer using shifted windows[C]// IEEE/CVF International Conference on Computer Vision (ICCV).Montreal,QC,Canada.IEEE, 2021:9992-10002.DOI:10.1109/ICCV48922.2021.00986.
Cited in this article [2]
[21]
HE K M, GKIOXARI G, DOLLÁR P, et al. Mask R-CNN[C]// IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas,NV,USA: IEEE, 2017:2961-2969.DOI:10.1109/ICCV.2017.322.
Cited in this article [1]
[22]
蒋开彬, 祝文娟, 潘文, 等. 基于SRAP标记的土沉香遗传多样性分析[J]. 中南林业科技大学学报, 2020, 40(1):131-136.
JIANG K B, ZHU W J, PAN W, et al. Analysis of genetic diversity of Aquilaria sinensis(Lour.) Gilg based on SRAP markers[J]. J Central South Univ For & Technol, 2020, 40(1):131-136.DOI:10.14067/j.cnki.1673-923x.2020.01.016.
Cited in this article [1]
[23]
庞圣江, 张培, 杨保国, 等. 林隙大小对土沉香人工更新幼树生长发育的影响[J]. 西北农林科技大学学报(自然科学版), 2020, 48(4):83-88.
PANG S J, ZHANG P, YANG B G, et al. Effects of gap size on growth of transplanted saplings of Aquilaria sinensis[J]. J Northwest A & F Univ (Nat Sci Ed), 2020, 48(4):83-88.DOI:10.13207/j.cnki.jnwafu.2020.04.011.
Cited in this article [1]
[24]
张小霞. 土沉香开发利用研究进展[J]. 防护林科技, 2020(4):63-66.
ZHANG X X. Research progress on development and utilization of Aquilaria sinensis[J]. Prot For Sci Technol, 2020(4):63-66.DOI:10.13601/j.issn.1005-5215.2020.04.024.
Cited in this article [1]
[25]
宋晓琛, 王西洋, 杨光, 等. 无机盐与激素混合对土沉香结香的诱导[J]. 林业科学, 2020, 56(8):121-130.
SONG X C, WANG X Y, YANG G, et al. Mechanism of agarwood formation under the induction of both inorganic salts and hormones[J]. Sci Silvae Sin, 2020, 56(8):121-130.DOI:10.11707/j.1001-7488.20200814.
Cited in this article [1]
[26]
洪仁辉, 尹吉锋, 陈彧, 等. 白木香重要害虫黄野螟研究进展[J]. 热带林业, 2019, 47(3):66-68.
HONG R H, YIN J F, CHEN Y, et al. Advances in research on the Heortia vitessoides Moore on Aquilaria sinensis[J]. Trop For, 2019, 47(3):66-68.DOI:10.3969/j.issn.1672-0938.2019.03.015.
Cited in this article [1]
[27]
王忠, 谢伟忠, 朱诚棋, 等. 黄野螟的羽化和生殖行为节律[J]. 中国森林病虫, 2018, 37(1):24-27,30.
WANG Z, XIE W Z, ZHU C Q, et al. Circadian rhythm of emergence and reproduction of Heortia vitessoides Moore (Lepidoptera:Crambidae)[J]. For Pest Dis, 2018, 37(1):24-27,30.
Cited in this article [1]
[28]
茅裕婷, 张蒙, 靳秀芳, 等. 土沉香对黄野螟的抗性研究[J]. 华南农业大学学报, 2017, 38(6):89-96.
MAO Y T, ZHANG M, JIN X F, et al. Study on resistance of Aquilaria sinensis against Heortia vitessoides[J]. J South China Agric Univ, 2017, 38(6):89-96.DOI:10.7671/j.issn.1001-411X.2017.06.014.
Cited in this article [1]
[29]
严珍, 岳建军. 温度及补充营养对黄野螟生长发育和繁殖的影响[J]. 热带作物学报, 2019, 40(9):1789-1795.
YAN Z, YUE J J. Effects of temperature and supplementary foods on the development and fecundity of Heortia vitessoides[J]. Chin J Trop Crops, 2019, 40(9):1789-1795.DOI:10.3969/j.issn.1000-2561.2019.09.017.
Cited in this article [1]
[30]
高新波, 莫梦竟成, 汪海涛, 等. 小目标检测研究进展[J]. 数据采集与处理, 2021, 36(3):391-417.
GAO X B, MO M J C, WANG H T, et al. Recent advances in small object detection[J]. J Data Acquis Process, 2021, 36(3):391-417.DOI:10.16337/j.1004-9037.2021.03.001.
Cited in this article [1]
[31]
LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]//IEEE Conference on Computer Vision and Pattern Recognition.Boston,MA, USA:IEEE, 2015:3431-3440.DOI:10.1109/CVPR.2015.7298965.
Cited in this article [2]
[32]
HINTON G E, SRIVASTAVA N, KRIZHEVSKY A, et al. Improving neural networks by preventing co-adaptation of feature detectors[EB/OL].[2022-12-01]. http://arxiv.org/abs/1207.0580.
http://arxiv.org/abs/1207.0580
Cited in this article [1]
[33]
张海燕, 徐心语, 马雪芬, 等. 超声图像中复合材料褶皱形态的Mask-RCNN识别方法[J]. 物理学报, 2022, 71(7):074302.
ZHANG H Y, XU X Y, MA X F, et al. Mask-RCNN recognition method of composite fold shape in ultrasound images[J]. Acta Phys Sin, 2022, 71(7):074302.DOI:10.7498/aps.71.20212009.
Cited in this article [1]
[34]
LIN T L, CHANG H Y, CHEN K H. Pest and disease identification in the growth of sweet peppers using faster R-CNN[J]. IEEE Int Conf Consumer Electron Taiwan (ICCE TW), 2019:1-2.DOI:10.1109/ICCE-TW46550.2019.8991893.
Cited in this article [1]
[35]
周维, 牛永真, 王亚炜, 等. 基于改进的YOLOv4-GhostNet水稻病虫害识别方法[J]. 江苏农业学报, 2022, 38(3):685-695.
ZHOU W, NIU Y Z, WANG Y W, et al. Rice pests and diseases identification method based on improved YOLOv4-GhostNet[J]. Jiangsu J Agr Sci, 2022, 38(3):685-695.DOI:10.3969/j.issn.1000-4440.2022.03.014.
Cited in this article [1]
[36]
刘晨曦, 刘大铭, 杨芳, 等. 基于改进水平集的水稻虫害分割算法[J]. 宁夏大学学报(自然科学版), 2019, 40(3):246-254.
LIU C X, LIU D M, YANG F, et al. Partitioning algorithm for rice pest based on improved level set method[J]. J Ningxia Univ (Nat Sci Ed), 2019, 40(3):246-254.DOI:10.3969/j.issn.0253-2328.2019.03.010.
Cited in this article [1]
[37]
卜俊怡, 孙国祥, 王迎旭, 等. 基于诱虫板图像的温室番茄作物害虫识别与监测方法[J]. 南京农业大学学报, 2021, 44(2):373-383.
BU J Y, SUN G X, WANG Y X, et al. Identification and monitoring method of tomato crop pests in greenhouse based on trapping board image[J]. J Nanjing Agric Univ, 2021, 44(2):373-383.DOI:10.7685/jnau.202005031.
Cited in this article [1]
PDF(3804 KB)

Accesses

Citation

Detail
Sections
Recommended
The full text is translated into English by AI, aiming to facilitate reading and comprehension. The core content is subject to the explanation in Chinese.

Author

Reader

Reviewer

www.nldxb.njfu.edu.cn

Edited ＆ Published by Editorial Department of Nanjing Forestry University(Natural Sciences Edition)
Address： No.159 Longpan Road,Nanjing 210037 Jiangsu,P.R.China
E-mail ：xuebao@njfu.edu.cn , xuebao@njfu.com.cn
Telephone +86-25-85428247,85427076
Distributed by China International Book Trading Corporation P.O. Box 399, Beijing ,P.R. China

/

〈 〉