UAV forestry land-cover image segmentation method based on attention mechanism and improved DeepLabV3+

doi:10.12302/j.issn.1000-2006.202209055

JOURNAL OF NANJING FORESTRY UNIVERSITY ›› 2024, Vol. 48 ›› Issue (4): 93-103.doi: 10.12302/j.issn.1000-2006.202209055

Special Issue: 专题报道Ⅲ：智慧林业之森林可视化研究

Previous Articles Next Articles

UAV forestry land-cover image segmentation method based on attention mechanism and improved DeepLabV3+

ZHAO Yugang¹(), LIU Wenping¹^,^*(), ZHOU Yan¹, CHEN Riqiang¹, ZONG Shixiang², LUO Youqing²

1. School of Information, Beijing Forestry University, Engineering Research Center for Forestry-oriented Intelligent Information Processing of National Forestry and Grassland Administration, Beijing 100083, China
2. School of Forestry, Beijing Forestry University, Beijing 100083, China

Received:2022-09-24 Revised:2022-11-01 Online:2024-07-30 Published:2024-08-05
Contact: LIU Wenping E-mail:15621377528@163.com;wendyl@vip.163.com

Abstract

Abstract:

【Objective】This study proposes the feature segmentation method Tree-DeepLab for unmanned aerial vehicle (UAV) forest images, based on an attention mechanism and the DeepLabV3+ semantic segmentation network, to extract the main feature distribution information in forest areas.【Method】First, the forest images were annotated according to feature types from six categories (Platanus orientalis, Ginkgo biloba, Populus sp., grassland, road, and bare ground) to obtain the semantic segmentation datasets. Second, the following improvements were made to the semantic segmentation network: (1) the Xception network, the backbone of the DeepLabV3+ semantic segmentation network, was replaced by ResNeSt101 with a split attention mechanism; (2) the atrous convolutions of different dilation rates in the atrous spatial pyramid pooling were connected using a combination of serial and parallel forms, while the combination of the atrous convolution dilation rates was simultaneously changed; (3) a shallow feature fusion branch was added to the decoder; (4) spatial attention modules were added to the decoder; and (5) efficient channel attention modules were added to the decoder.【Result】Training and testing were performed based on an in-house dataset. The experimental results revealed that the Tree-DeepLab semantic segmentation model had mean pixel accuracy (mPA) and mean intersection over union (mIoU) values of 97.04% and 85.01%, respectively, exceeding those of the original DeepLabV3+ by 4.03 and 14.07 percentage points, respectively, and outperforming U-Net and PSPNet.【Conclusion】The study demonstrates that the Tree-DeepLab semantic segmentation model can effectively segment UAV aerial photography images of forest areas to obtain the distribution information of the main feature types in forest areas.

Key words: unmanned aerial vehicle(UAV), land-cover image segmentation, forestry images, DeepLabV3+, attention mechanism, ResNeSt

CLC Number:

S758
TP391

ZHAO Yugang, LIU Wenping, ZHOU Yan, CHEN Riqiang, ZONG Shixiang, LUO Youqing. UAV forestry land-cover image segmentation method based on attention mechanism and improved DeepLabV3+[J]. JOURNAL OF NANJING FORESTRY UNIVERSITY, 2024, 48(4): 93-103.

Figures/Tables 12

Fig. 1

Fig. 2

Fig. 3

Fig. 4

Fig. 5

Fig. 6

Fig. 7

Fig. 8

Table 1

Table 2

Table 3

Fig. 9

References 31

[1]	王静, 高建中. 林地地块特征对农户林业生产效率的影响[J]. 林业经济问题, 2021, 41(6):577-582.
	WANG J, GAO J Z. The effects of the characteristics of forest land parcels on farmers’ forestry production efficiency[J]. News For Econ, 2021, 41(6):577-582. DOI: 10.16832/j.cnki.1005-9709.20210072.
[2]	DALPONTE M, ØRKA H O, GOBAKKEN T, et al. Tree species classification in boreal forests with hyperspectral data[J]. IEEE Trans Geosci Remote Sens, 2013, 51(5):2632-2645. DOI: 10.1109/TGRS.2012.2216272.
[3]	BLANCO S R, HERAS D B, ARGÜELLO F. Texture extraction techniques for the classification of vegetation species in hyperspectral imagery: bag of words approach based on superpixels[J]. Remote Sens, 2020, 12(16):2633. DOI: 10.3390/rs12162633.
[4]	THANH NOI P, KAPPAS M. Comparison of random forest, k-nearest neighbor, and support vector machine classifiers for land cover classification using sentinel-2 imagery[J]. Sensors, 2017, 18(1):18. DOI: 10.3390/s18010018.
[5]	YUAN Y, HU X Y. Random forest and objected-based classification for forest pest extraction from UAV aerial imagery[J]. Int Arch Photogramm Remote Sens Spatial Inf Sci, 2016, XLI-B1:1093-1098. DOI: 10.5194/isprs-archives-xli-b1-1093-2016.
[6]	赵庆展, 江萍, 王学文, 等. 基于无人机高光谱遥感影像的防护林树种分类[J]. 农业机械学报, 2021, 52(11):190-199.
	ZHAO Q Z, JIANG P, WANG X W, et al. Classification of protection forest tree species based on UAV hyperspectral data[J]. Trans Chin Soc Agric Mach, 2021, 52(11):190-199. DOI: 10.6041/j.issn.1000-1298.2021.11.020.
[7]	戴鹏钦, 丁丽霞, 刘丽娟, 等. 基于FCN的无人机可见光影像树种分类[J]. 激光与光电子学进展, 2020, 57(10):36-45.
	DAI P Q, DING L X, LIU L J, et al. Tree species identification based on FCN using the visible images obtained from an unmanned aerial vehicle[J]. Laser Optoelectron Prog, 2020, 57(10):36-45. DOI: 10.3788/LOP57.101001.
[8]	张军国, 冯文钊, 胡春鹤, 等. 无人机航拍林业虫害图像分割复合梯度分水岭算法[J]. 农业工程学报, 2017, 33(14):93-99.
	ZHANG J G, FENG W Z, HU C H, et al. Image segmentation method for forestry unmanned aerial vehicle pest monitoring based on composite gradient watershed algorithm[J]. Trans Chin Soc Agric Eng, 2017, 33(14):93-99. DOI: 10.11975/j.issn.1002-6819.2017.14.013.
[9]	张增, 王兵, 伍小洁, 等. 无人机森林火灾监测中火情检测方法研究[J]. 遥感信息, 2015, 30(1):107-110, 124.
	ZHANG Z, WANG B, WU X J, et al. An algorithm of forest fire detection based on UAV remote sensing[J]. Remote Sens Inf, 2015, 30(1):107-110, 124. DOI: 10.3969/j.issn.1000-3177.2015.01.018.
[10]	刘文萍, 仲亭玉, 宋以宁. 基于无人机图像分析的树木胸径预测[J]. 农业工程学报, 2017, 33(21):99-104.
	LIU W P, ZHONG T Y, SONG Y N. Prediction of trees diameter at breast height based on unmanned aerial vehicle image analysis[J]. Trans Chin Soc Agric Eng, 2017, 33(21):99-104. DOI: 10.11975/j.issn.1002-6819.2017.21.012.
[11]	MARTINS J, JUNIOR J M, MENEZES G, et al. Image segmentation and classification with SLIC superpixel and convolutional neural network in forest context[C]// IGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Symposium. Yokohama, Japan: IEEE, 2019:6543-6546. DOI: 10.1109/IGARSS.2019.8898969.
[12]	LONG J, SHELHAMER E, DARRELL T. Fully convolutional networks for semantic segmentation[C]// 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Boston, MA, USA: IEEE, 2015:3431-3440. DOI: 10.1109/CVPR.2015.7298965.
[13]	RONNEBERGER O, FISCHER P, BROX T. U-net: Convolutional networks for biomedical image segmentation[C]// International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer, 2015:234-241. DOI: 10.1007/978-3-319-24574-4_28.
[14]	ZHAO H S, SHI J P, QI X J, et al. Pyramid scene parsing network[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA: IEEE, 2017:6230-6239. DOI: 10.1109/CVPR.2017.660.
[15]	LIN G S, MILAN A, SHEN C H, et al. RefineNet: multi-path refinement networks for high-resolution semantic segmentation[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA: IEEE, 2017:5168-5177. DOI: 10.1109/CVPR.2017.549.
[16]	CHEN L C, PAPANDREOU G, KOKKINOS I, et al. Semantic image segmentation with deep convolutional nets and fully connected CRFs[DB/OL]. (2014-09-14)[2022-05-25]. https://arXiv.org/abs/1412.7062. DOI: 10.48550/arXiv.1412.7062.
[17]	CHEN L C, PAPANDREOU G, KOKKINOS I, et al. DeepLab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs[J]. IEEE Trans Pattern Anal Mach Intell, 2018, 40(4):834-848. DOI: 10.1109/TPAMI.2017.2699184.
[18]	CHEN L C, PAPANDREOU G, SCHROFF F, et al. Rethinking atrous convolution for semantic image segmentation[DB/OL]. (2017-07-19)[2022-05-05]. https://arXiv.org/abs/1706.05587. DOI: 10.48550/arXiv.1706.05587.
[19]	CHEN L C, ZHU Y K, PAPANDREOU G, et al. Encoder-decoder with atrous separable convolution for semantic image segmentation[C]// European Conference on Computer Vision. Cham: Springer, 2018:833-851. DOI: 10.1007/978-3-030-01234-2_49.
[20]	韩蕊, 慕涛阳, 赵伟, 等. 基于无人机多光谱影像的柑橘树冠分割方法研究[J]. 林业工程学报, 2021, 6(5):147-153.
	HAN R, MU T Y, ZHAO W, et al. Research on citrus canopy segmentation method based on UAV multispectral image[J]. Journal of Forestry Engineering, 2021, 6(5):147-153. DOI: 10.13360/j.issn.2096-1359.202011021.
[21]	刘文定, 田洪宝, 谢将剑, 等. 基于全卷积神经网络的林区航拍图像虫害区域识别方法[J]. 农业机械学报, 2019, 50(3):179-185.
	LIU W D, TIAN H B, XIE J J, et al. Identification methods for forest pest areas of UAV aerial photography based on fully convolutional networks[J]. Trans Chin Soc Agric Mach, 2019, 50(3):179-185. DOI: 10.6041/j.issn.1000-1298.2019.03.019.
[22]	徐辉, 祝玉华, 甄彤, 等. 深度神经网络图像语义分割方法综述[J]. 计算机科学与探索, 2021, 15(1):47-59.
	XU H, ZHU Y H, ZHEN T, et al. Survey of image semantic segmentation methods based on deep neural network[J]. J Front Comput Sci Technol, 2021, 15(1):47-59. DOI: 10.3778/j.issn.1673-9418.2004039.
[23]	HU J, SHEN L, SUN G. Squeeze-and-excitation networks[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018:7132-7141. DOI: 10.1109/CVPR.2018.00745.
[24]	WOO S, PARK J, LEE J Y, et al. CBAM: convolutional block attention module[M]// Computer Vision-ECCV 2018. Cham: Springer International Publishing, 2018:3-19. DOI: 10.1007/978-3-030-01234-2_1.
[25]	WANG Q L, WU B G, ZHU P F, et al. ECA-net: efficient channel attention for deep convolutional neural networks[C]// 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, WA, USA: IEEE, 2020:11531-11539. DOI: 10.1109/CVPR42600.2020.01155.
[26]	ZHANG H, WU C R, ZHANG Z Y, et al. ResNeSt: split-attention networks[C]// 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). New Orleans, LA, USA: IEEE, 2022:2735-2745. DOI: 10.1109/CVPRW56347.2022.00309.
[27]	CHOLLET F. Xception: deep learning with depthwise separable convolutions[C]// 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI, USA: IEEE, 2017:1800-1807. DOI: 10.1109/CVPR.2017.195.
[28]	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]// 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Las Vegas, NV, USA: IEEE, 2016:770-778. DOI: 10.1109/CVPR.2016.90.
[29]	YANG M K, YU K, ZHANG C, et al. DenseASPP for semantic segmentation in street scenes[C]// 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, UT, USA: IEEE, 2018:3684-3692. DOI: 10.1109/CVPR.2018.00388.
[30]	李丹, 张俊杰, 赵梦溪. 基于FCM和分水岭算法的无人机影像中林分因子提取[J]. 林业科学, 2019, 55(5):180-187.
	LI D, ZHANG J J, ZHAO M X. Extraction of stand factors in UAV image based on FCM and watershed algorithm[J]. Sci Silvae Sin, 2019, 55(5):180-187. DOI: 10.11707/j.1001-7488.20190520.
[31]	刘旭光, 肖啸, 兰玉彬, 等. 应用可见光遥感影像的林区植被分割方法[J]. 东北林业大学学报, 2023, 51(4):62-67.
	LIU X G, XIAO X, LAN Y B, et al. Forest vegetation segmentation method with UAV visible light remote sensing images[J]. Journal of Northeast Foresrty University, 2023, 51(4):62-67. DOI: 10.13759/j.cnki.dlxb.2023.04.008.

Metrics

Comments

www.nldxb.njfu.edu.cn

Edited ＆ Published by Editorial Department of Nanjing Forestry University(Natural Sciences Edition)
Address： No.159 Longpan Road,Nanjing 210037 Jiangsu,P.R.China
E-mail ：xuebao@njfu.edu.cn , xuebao@njfu.com.cn
Telephone +86-25-85428247,85427076
Distributed by China International Book Trading Corporation P.O. Box 399, Beijing ,P.R. China

模型 model	原始ASPP original ASPP	D-ASPP 模块	SP-ASPP 模块	ASPP 扩张率组合 ASPP dilation rate combinations	模型参数/MB model parameters	训练时间/min training time	测试时间/s testing time	平均像素精度/% mPA	平均交并比/% mIoU
1	√			(6,12,18)	49.41	259	23.83	95.82	78.24
2		√		(6,12,18,24)	50.43	330	23.91	96.74	81.69
3		√		(3,6,9,12)	50.43	333	23.66	96.81	82.01
4			√	(6,12,18,24)	48.94	271	25.51	96.63	82.41
5			√	(3,6,9,12)	48.94	252	22.62	97.02	83.18

模型 model	主干网络 backbone	原始ASPP original ASPP	SP-ASPP 模块	浅层特征融合分支 shallow feature fusion branche	SA 模块	ECA 模块	训练时间/min training time	测试时间/s testing time	平均像素精度/% mPA	平均交并比/% mIoU
1	Xception	√					256	22.43	93.01	70.94
2	ResNet101	√					237	19.41	95.27	75.52
3	ResNeSt101	√					259	23.83	95.82	78.24
4	ResNeSt101	√		√			268	23.72	96.78	82.13
5	ResNeSt101		√				252	22.62	97.02	83.18
6	ResNeSt101		√	√			261	23.23	96.94	83.96
7	ResNeSt101		√	√	√		270	23.97	97.01	84.02
8	ResNeSt101		√	√		√	264	23.44	97.05	84.13
9	ResNeSt101		√	√	√	√	272	24.79	97.04	85.01

语义分割模型 semantic segmentation models	平均像素精度/% mPA	训练时间/min training time	测试时间/s testing time	平均交并比/% mIoU	交并比/% IoU
语义分割模型 semantic segmentation models	平均像素精度/% mPA	训练时间/min training time	测试时间/s testing time	平均交并比/% mIoU	杨树 Populus sp.	银杏 Ginkgo biloba	法国梧桐 Platanus orientalis	道路 road	草地 grassland	裸地 bare groud
U-Net	94.01	241	20.25	67.50	74.67	93.99	90.87	66.53	33.31	45.62
PSPNet	93.28	608	42.93	71.59	80.18	94.18	88.37	51.94	44.17	70.69
DeepLabV3+	93.01	256	22.43	70.94	84.67	93.12	82.05	55.75	48.32	61.74
Tree-DeepLab	97.04	272	24.79	85.01	92.55	97.01	91.95	81.53	68.64	78.35

UAV forestry land-cover image segmentation method based on attention mechanism and improved DeepLabV3+

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 31

Related Articles 5

Recommended Articles

Metrics

Comments

Author

Reader

Reviewer

[1]	NIU Hongjian, LIU Wenping, CHEN Riqiang, ZONG Shixiang, LUO Youqing. Dehaze algorithm for woodland UAV images based on Resnet [J]. JOURNAL OF NANJING FORESTRY UNIVERSITY, 2024, 48(2): 175-18.
[2]	YANG Kun, FAN Xijian, BO Weihao, LIU Jie, WANG Junling. Plant disease and pest detection based on visiual attention enhancement [J]. JOURNAL OF NANJING FORESTRY UNIVERSITY, 2023, 47(3): 11-18.
[3]	CHENG Jianbin,WANG Jibin, WANG Nianjin,YAO Xiaohua. The fruit-xenia effects of Carya illinoensis pollination on C. cathayensis assisted by unmanned aerial vehicle(UAV) [J]. JOURNAL OF NANJING FORESTRY UNIVERSITY, 2019, 43(04): 199-202.
[4]	TAO Huan, LI Cunjun, XIE Chunchun, ZHOU Jingping, HUAI Heju, JIANG Liya, LI Fengtao. Recognition of red-attack pine trees from UAV imagery based on the HSV threshold method [J]. JOURNAL OF NANJING FORESTRY UNIVERSITY, 2019, 43(03): 99-106.
[5]	DAI Tingting, MA Jun, XU Yannan. Application of unmanned aerial vehicle(UAV)image automatic stitching in landscape planning based on Agisoft PhotoScan [J]. JOURNAL OF NANJING FORESTRY UNIVERSITY, 2018, 42(04): 165-170.