Proposing a model to predict relatedness between knowledge units in programming question-answering websites using deep learning techniques: a case study of Stack

Document Type : Original Article

Authors

Faculty of Information Technology and Computer Engineering, Azarbaijan Shahid Madani University, Tabriz, Iran.

Abstract

The Stack Overflow website is one of the most popular communities used by millions of programmers. If we consider a question and its corresponding answers as a knowledge unit on the Stack Overflow website, then there are different semantic relationships between two knowledge units, which include duplicate, direct, and indirect relationships with the proposed question. Recognizing different categories of semantic relationship between knowledge units in Stack Overflow can significantly improve the effectiveness and efficiency of information search. In this study, a hybrid approach based on deep learning methods and traditional similarity criteria is presented to detect the relationship between questions. In particular, two deep network architectures are presented, the first architecture consists of a long short-term memory network as well as a cosine similarity calculation layer. The second architecture is an extension of the first architecture by adding an attention mechanism. The proposed approach was evaluated on a dataset of Java programming language contining 40000 questions. The obtained results show that in terms of F1, Recall and Precision, the proposed model performs better than the existing models. Specifically, the model proposed in this article has a 17.3% improvement in terms of F1 measure compared to the best current model. Also, the results of the experiments show that using the pre-trained word embedding model significantly improves the performance of the presented models.

Keywords

References

[1] مهدی دهقان، احمدعلی آبین، « بازیابی و رتبه‌بندی افراد خبره با استفاده از مدل ترجمه مبتنی بر خوشه‌بندی»، مجله مهندسی برق دانشگاه تبریز، جلد 49، شماره 3، صفحات 1106-1095، 1398.
[2]  P. Chakraborty, R. Shahriyar, A. Iqbal, and G. Uddin, "How do developers discuss and support new programming languages in technical Q&A site? An empirical study of Go, Swift, and Rust in Stack Overflow", Information and Software Technology, vol. 137, pp. 106603, 2021.
[3]  H. Shu, P. Gao, Z. Yang, C. Li, and M. Wu, "Exploring the Feasibility of Transformer Based Models on Question Relatedness", Proceedings of 2022 IEEE 24th Int Conf on High Performance Computing & Communications; 8th Int Conf on Data Science & Systems; 20th Int Conf on Smart City; 8th Int Conf on Dependability in Sensor, Cloud & Big Data Systems & Application (HPCC/DSS/SmartCity/DependSys), pp. 831-838 , 2022.
[4]  J. He, Z. Xin, B. Xu, T. Zhang, K. Kim, Z. Yang, et al., "Representation Learning for Stack Overflow Posts: How Far are We?", arXiv preprint arXiv:2303.06853, 2023.
[5]  A. Kumar, D. Ghadiyali, S. Chimalakonda, and A. S. M. Venigalla, "SOCluster-Towards Answering Unanswered Questions on Stack Overflow via Answered Questions", Proceedings of the 16th Innovations in Software Engineering Conference, pp . 1-5,2023.
[6]  S.-K. Guo, S.-W. Wang, H. Li, Y.-L. Fan, Y.-Q. Liu, and B. Zhang, "Multi-Feature Fusion Based Structural Deep Neural Network for Predicting Answer Time on Stack Overflow", Journal of Computer Science and Technology, vol. 38, no. 3, pp. 582-599, 2023.
[7]  P. K. Roy, S. Saumya, J. P. Singh, S. Banerjee, and A. Gutub, "Analysis of community question‐answering issues via machine learning and deep learning: State‐of‐the‐art review", CAAI Transactions on Intelligence Technology, vol. 8, no. 1, pp. 95-117, 2023.
[8] B. Xu, D. Ye, Z. Xing, X. Xia, G. Chen, and S. Li, "Predicting semantically linkable knowledge in developer online forums via convolutional neural network", Proceedings of 2016 31st IEEE/ACM International Conference on Automated Software Engineering (ASE), pp. 51-62, 2016.
[9] L. Ponzanelli, G. Bavota, M. Di Penta, R. Oliveto, and M. Lanza, "Prompter: Turning the IDE into a self-confident programming assistant", Empirical Software Engineering, vol. 21, pp. 2190-2231, 2016.
[10] Y. Zhang, D. Lo, X. Xia, and J.-L. Sun, "Multi-factor duplicate question detection in stack overflow", Journal of Computer Science and Technology, vol. 30, pp. 981-997, 2015.
[11] W. E. Zhang, Q. Z. Sheng, J. H. Lau, E. Abebe, and W. Ruan, "Duplicate detection in programming question answering communities", ACM Transactions on Internet Technology (TOIT), vol. 18, no. 3, pp. 1-21, 2018.
[12] W. Gao, J. Wu, and G. Xu, "Detecting duplicate questions in stack overflow via source code modeling", Int J Software Eng Knowl Eng, vol. 32, no. 02, pp. 227-255, 2022.
[13] B. Xu, A. Shirani, D. Lo, and M. A. Alipour, "Prediction of relatedness in stack overflow: deep learning vs. svm: a reproducibility study", Proceedings of the 12th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, pp. 1-10, 2018.
[14] A. Shirani, B. Xu, D. Lo, T. Solorio, and A. Alipour, "Question relatedness on stack overflow: the task, dataset, and corpus-inspired models", arXiv preprint arXiv:1905.01966, 2019.
[15] رضا خدایی، محمدعلی بالافر، سیدناصر رضوی،« اثربخشی بسط پرس‌وجودمبتنی بر خوشه‌بندی اسناد شبه‌بازخورد با الگوریتم»، مجله مهندسی برق دانشگاه تبریز، جلد 46، شماره 1، صفحات 143-151، 1395.
[16] مرضیه رحیمی، عرفان جلیلی جلال، حسین رحیمی، « تولید کلمات کلیدی متون فارسی با استفاده از یادگیری انتقالی»، مجله مهندسی برق دانشگاه تبریز، جلد 52، شماره 2، صفحات 115-123، 1401 .
[17] E. Zafarani-Moattar, M. R. Kangavari, and A. M. Rahmani, "Topic Detection on COVID-19 Tweets: A Comparative Study on Clustering and Transfer Learning Models", TABRIZ JOURNAL OF ELECTRICAL ENGINEERING, vol. 52, no. 4, pp. 281-291, 2022.
[18] M.-T. Luong, H. Pham, and C. D. Manning, "Effective approaches to attention-based neural machine translation", arXiv preprint arXiv:1508.04025, 2015.
[19] M. Ahasanuzzaman, M. Asaduzzaman, C. K. Roy, and K. A. Schneider, "Mining duplicate questions in stack overflow", Proceedings of the 13th International Conference on Mining Software Repositories, pp. 402-412, 2016.
[20] L. Wang, L. Zhang, and J. Jiang, "Duplicate Question Detection With Deep Learning in Stack Overflow", IEEE Access, vol. 8, pp. 25964-25975, 2020.
[21] R. F. G. Silva, K. Paixão, and M. d. A. Maia, "Duplicate question detection in stack overflow: A reproducibility study", Proceedings of 2018 IEEE 25th International Conference on Software Analysis, Evolution and Reengineering (SANER), pp. 572-581, 2018.
[22] W. E. Zhang, Q. Z. Sheng, Y. Shu, and V. K. Nguyen, "Feature analysis for duplicate detection in programming QA communities", Proceedings of Advanced Data Mining and Applications: 13th International Conference, ADMA 2017, pp. 623–638, 2017.
[23] N. Ansari and R. Sharma, "Identifying semantically duplicate questions using data science approach: A quora case study", arXiv preprint arXiv:2004.11694, 2020.
[24] A. Kamienski, A. Hindle, and C.-P. Bezemer, "Analyzing Techniques for Duplicate Question Detection on Q&A Websites for Game Developers", Empirical Software Engineering, vol. 28, no. 1, pp. 17, 2023.
[25] D. Charlet and G. Damnati, "Simbow at semeval-2017 task 3: Soft-cosine semantic similarity between questions for community question answering," Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pp. 315–319, 2017.
[26] J. Pei, Y. Wu, Z. Qin, Y. Cong, and J. Guan, "Attention-based model for predicting question relatedness on Stack Overflow", Proceedings of. 2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR), pp. 97-107, 2021.
[27] "Stack Overflow", avaiable online at: https://en.wikipedia.org/wiki/Stack_Overflow
[28] J. Pennington, R. Socher, and C. D. Manning, "Glove: Global vectors for word representation", Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
[29] A. Darvish and S. Shamekhi, "A hybrid multi-scale CNN-LSTM deep learning model for the identification of protein-coding regions in DNA sequences", TABRIZ JOURNAL OF ELECTRICAL ENGINEERING, vol. 52, no. 2, pp. 137-146, 2022.
[30] F. Chollet, "Keras: The python deep learning library", Astrophysics source code library, 2018.
[31] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization", arXiv preprint arXiv:1412.6980, 2014.
TABRIZ JOURNAL OF ELECTRICAL ENGINEERING

Articles in Press, Corrected Proof
Available Online from 14 October 2023

Files

Share

How to cite

Statistics