| Analysis of ChatGPT as a Question-Answering Tool |
| Pit Pichappan, M. Krishnamurthy, P. Vijayakumar Digital Information Research Labs Chennai 600017, India., Documentation Research & Training Center Indian Statistical Institute, Bangalore 5600059, India., Central University, Pondicherry 6050105 India |
| Abstract: ChatGPT, in recent months, has made a significant impact and exposure in the information world. Many studies have been conducted within a shorter timeframe about its efficiency, reliability, ethics, accuracy and acceptance. Besides, hundreds of opinions and perception-based analyses have also emerged. In this work, we look at the ChatGPT as a question-answering tool. We have used randomly generated prompts to solicit answers and analysed the results from a text analysis angle. The answers are compared with text analysers both manually and statistically. ChatGPT still needs more precision for linguistic effects and fails to meet comprehensive users’ requirements. |
| Keywords: ChatGPT, Question-answering tools, Text Analysis, AI tools, Natural Language Analysis Analysis of ChatGPT as a Question-Answering Tool |
| DOI:https://doi.org/10.6025/jdim/2023/21/2/50-61 |
| Full_Text PDF 1.60 MB Download: 17 times |
| References: [1] https://fireflies.ai/blog/generative-ai-or-gpt-3-apps [2] Number of parameters in notable Artificial Intelligence Systems, Our World in Data. [Online]. Available: https://ourworldindata.org/grapher/artificial-intelligence-parametercount. [Accessed: 22-Mar-2023]. [3] Hughes, A. (2023). CHATGPT: Everything you need to know about OpenAI’s GPT-4 tool, ChatGPT: Everything you need to know about OpenAI’s GPT-4 upgrade — BBC Science, Focus Magazine, 16-Mar-2023. [Online]. Available: https://www.sciencefocus.com/futuretechnology/gpt-3 [Accessed: 22-Mar-2023]. [4] https://towardsdatascience.com/gpt-3-a-complete-overview-190232eb25fd [5] Yu, Caiyang., Liu, Xianggen., Tang, Chenwei., Feng, Wentao., Jiancheng GPT-NAS: Neural Architecture Search with the Generative Pre-Trained Model. Lv.arXiv:2305.05351v1 [cs.CV] 9 May 2023] [6] Hendy, A., Abdelrehim, M., Sharaf, A., Raunak, V., Gabr, M., Matsushita, H., Kim, Y. J., Afify, M., Awadalla, H. H. (2023). How good are gpt models at machine translation? a comprehensive evaluation, arXiv preprint arXiv:2302.09210, 2023 [7] Mathew L. Bindu, V. (2020). A review of natural language processing techniques for sentiment analysis using pre-trained models, In: 2020 Fourth International Conference on Computing Methodologies and Communication (ICCMC). IEEE, 2020, p. 340–345. [8] Bongini, P., Becattini, F., Del Bimbo, A. (2022). Is gpt-3 all you need for visual question answering in cultural heritage? In: Computer Vision–ECCV 2022 Workshops: Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part I. Springer, 2023, p. 268–281. [9] Limna, Pongsakorn., Kraiwanit, Tanpat., Jangjarat, Kris., KlayklungPrapasiri, Chocksathaporn, Piyawatjana (2023). The use of ChatGPT in the digital era: Perspectives on chatbot implementation, Journal of Applied Learn- [10] Cooper, K. (2021, November 1). OpenAI GPT-3: Ev- [11] Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Amodei, D. (2020). Language models are few-shot learners. Advances in Neural Information Processing Systems, 33, 1877-1901. [12] Dehouche, N. (2021). Plagiarism in the age of massive Generative Pre-trained Transformers (GPT-3). Ethics in Science and Environmental Politics, 21, 17-23 [13] Fyfe, P. (2022). How to cheat on your final paper: Assigning AI for student writing. AI & Society, 1-11. [14] Anson, C. M., Straume, I. (2022). Amazement and trepidation: Implications of AI-based natural language production for the teaching of writing. Journal of Academic Writing, 12 (1) 1-9. [15] Anson, C. M. (2022). AI-based text generation and the social construction of “fraudulent authorship”: A revisitation. Composition Studies, 50, 37-46. [16] Köbis, N., Mossink, L. D. (2021). Artificial intelligence versus Maya Angelou: Experimental evidence that people cannot differentiate AI-generated from human-written poetry. Computers in Human Behavior, 114, 106553 [17] Tate, T., Doroudi, S., Ritchie, D., Xu, Y. (2023). Educational research and AI-generated writing: Confronting the coming tsunami. Pre-print. https://edarxiv.org/4mec3 [18] Moore, S., Nguyen, H. A., Bier, N., Domadia, T., Stamper, J. (2022). Assessing the quality of student-generated short answer questions using GPT-3. In: European Conference on Technology Enhanced Learning (p. 243-257). Springer. [19] Elkins, K., Chun, J. (2020). Can GPT-3 pass a writer’s Turing test? Journal of Cultural Analytics, 5 (2) 17212. [20] Nguyen, H. A., Bhat, S., Moore, S., Bier, N., Stamper, J. (2022). Towards generalized methods for automatic question generation in educational domains. In: European Conference on Technology Enhanced Learning p. 272-284. Springer. [21] Yu, Caiyang., Liu, Xianggen., Tang, Chenwei., Feng, Wentao., Lv, Jiancheng.(GPT-NAS: Neural Architecture Search with the Generative Pre-Trained Model.arXiv:2305.05351v1 [cs.CV] 9 May 2023 [22] Hendy, A., Abdelrehim, M., Sharaf, A., Raunak, V., Gabr, M., Matsushita, H., Kim, Y. J., Afify, M. Awadalla, H. H (2023). How good are gpt models at machine trans- lation? a comprehensive evaluation, arXiv preprint arXiv:2302.09210 [23] Mathew, L., Bindu, V (2020). A review of natural language processing techniques for sentiment analysis using pre-trained models, In: 2020 Fourth International Conference on Computing Methodologies and Communication (ICCMC). IEEE, 2020, p. 340–345. [24] Bongini, P., Becattini, F. Del Bimbo, A (2022). Is gpt-3 all you need for visual question answering in cultural heritage? In: Computer Vision–ECCV 2022 Workshops: Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part I. Springer, 2023, p. 268–281. [25] Li, Chenglu., Xing, Wanli. (2021). Natural Language Generation Using Deep Learning to Support MOOC Learners. International Journal of Artificial Intelligence in Education v. 31, p.186–214. [26] Caballé, S., Conesa, J. (2019). Conversational Agents in Support for Collaborative Learning in MOOCs: An Analytical Review. In F. Xhafa, L. Barolli, & M. Greguš (Eds.), Advances in Intelligent Networking and Collaborative Systems. INCoS 2018. Lecture notes on data engineering and communications technologies (Vol. 23). Cham: Springer. [27] Kumar, R., Rose, C. P. (2010). Architecture for building conversational agents that support collaborative learning. IEEE Transactions on Learning Technologies, 4 (1) 21–34. https://doi.org/10.1109/TLT.2010.41 [28] Mittal, A., Vigentini, L., Djatmiko, M., Prusty, G., Sharma, Y., King, M. E. (2018). MOOC-O-Bot: Using Cognitive Technologies to Extend Knowledge Support in MOOCs. Proceedings of the 2018 IEEE International Conference on Teaching, Assessment, and Learning for Engineering (TALE), 69–76. https://doi.org/10.1109/tale.2018.8615453.] [29] Shawar and Atwell Abu Shawar, B., Atwell, E. (2007). Chatbots: Are They Really Useful? LDV-Forum Band, 22, 29-49. [30] Indurthi, S. R., Raghu, D., Khapra, M. M., Joshi, S. (2017). Generating natural language question-answer pairs from a knowledge graph using a RNN-based question generation model. In: M. Lapata, P. Blunsom, A. Koller (Eds), Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers (pp. 376-385). Association for Computational Linguistics. https://doi.org/10.18653/v1/e17-1036. [31] Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I. (2019). Language models are unsupervised multitask learners. OpenAI Blog, 1 (8) 9. Retrieved 4 Jan 2020, from http://www.persagen.com/files/misc/ radford2019language.pdf [32] Zhang, Y., Sun, S., Galley, M., Chen, Y.-C., Brockett, C., Gao, X., Dolan, B. (2020). DialoGPT: Large-scale generative pre-training for conversational response generation. In: A. Celikyilmaz, T. Wen (Eds), Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (p. 270–278). Association for Computational Linguistics. https://doi.org/10.18653/v1/2020.acl-demos.30 [33] Eva A. M., van Dis, Bollen, Johan., Rooij, Robert van., Zuidema, Willem., Bockting, Claudi L. (2023). ChatGPT: five priorities for research, Nature V. 614. 224-2266. 9 February 2023. [34] Rich, A. S., Gureckis, T. M.(2019). Nature Mach. Intell. 1, 174–180. 2019. [35] Else, Holly., Gao, C. A. et al (2022). ABSTRACTS WRITTEN BY CHATGPT FOOL SCIENTISTS by. Nature V. 613, 19 January 2023 p. 423. Preprint at bioRxiv https://doi.org/10.1101/2022.12.23.521610. [36] ChatGPT: five priorities for research by Eva A. M. van Dis, Johan Bollen, Robert van Rooij, Willem Zuidema & Claudi L. Bockting- 224-2266, Nature, Vol 614 . 9 February 2023. [37] JOUR Kenney, Noah A Brief Analysis of the Architecture, Limitations, and Impacts of ChatGPT DO - 10.5281/zenodo.7762245 2023/03/23 [38] Zhu, Yaoming., Lu, Sidi., Zheng, Lei., Guo, Jiaxian., Zhang, Weinan., Wang, Jun ., Yu, Yong. (2018). Texygen: A Benchmarking Platform for Text Generation Models. ACM SIGIR’18, July 8-12, 2018, p. 1097-1100. Ann Arbor, MI, USA [39] Sallam, Malik. (2023). ChatGPT Utility in Healthcare Education, Research, and Practice: Systematic Review on the Promising Perspectives and Valid Concerns, Healthcare, 11 (6) 887; https://doi.org/10.3390/healthcare11060887
|