A United States Purdue University study found that OpenAI developed an artificial intelligence chatbot ChatGPT in answering software programming questions, with an error rate of more than half, and can fool one-third of the questioner.
The research team analyzed ChatGPT’s answers to 517 questions on the Stack Overflow website, evaluating them for correctness, consistency, comprehensiveness, and brevity. They also analyzed the responses for language and sentiment and administered a questionnaire to 12 volunteer participants. The study concluded that while the ChatGPT responses had a good linguistic style, 52% of the responses were incorrect and 77% were lengthy. Participants were only able to recognize errors in their responses if they were obvious. Otherwise, they were misled by ChatGPT’s friendly, authoritative, and detailed language style.
Stack Overflow is a website dedicated to Q&A for programmers and is considered a great source for developers to get code snippets and solve problems. However, the site has also faced declining traffic, declining content quality and declining user trust in recent years. According to an April report from SimilarWeb, the site’s monthly traffic has dropped 6 percent since January 2022 and 13.9% in March, and ChatGPT may be contributing to its traffic decline. Stack Overflow, on the other hand, has denied SimilarWeb’s assessment, stating that its traffic has been driven by search algorithms and the Stack Overflow denied SimilarWeb’s assessment, saying that its traffic was influenced by its search algorithm and a surge in interest in ChatGPT.
OpenAI admits on its ChatGPT website that its software “may produce inaccurate information about people, places, or facts.” The company has not yet responded to the Purdue study.
