ChatGPT and other big language models do well on financial license tests when selecting multiple-choice responses, but they struggle on more complex problems.
In a research sponsored by Washington State University, the artificial intelligence language models BARD, Llama, and ChatGPT examined over 10,000 answers to financial test questions.
In addition to selecting responses, the researchers asked the models to justify their choices. They then contrasted the text responses with those provided by human experts. Even while two ChatGPT versions excelled at these tasks, they continued to exhibit a significant degree of inaccuracy when discussing more complex subjects.
“It’s far too early to be worried about ChatGPT taking finance jobs completely,” said study author DJ Fairhurst of WSU’s Carson College of Business. “For broad concepts where there have been good explanations on the internet for a long time, ChatGPT can do a very good job at synthesizing those concepts. If it’s a specific, idiosyncratic issue, it’s really going to struggle.”
Fairhurst and co-author Daniel Greene of Clemson University used questions from the Securities Industry Essentials test and the Series 6, 7, 65, and 66 license examinations for their study, which was published in the Financial Analysts Journal.
The researchers requested textual explanations from the AI models in order to go beyond their capacity to just select the correct response. Additionally, they selected questions based on certain work duties that financial professionals may really carry out.
“Passing certification exams is not enough. We really need to dig deeper to get to what these models can really do,” said Fairhurst.
The model that performed the best out of all of them was ChatGPT’s version 4.0, which was compensated and produced responses that were most like those of human experts. Additionally, compared to the other models, its accuracy was 18 to 28 percentage points greater. This was altered, though, when the researchers improved ChatGPT 3.5, the previous free version, by providing it with samples of appropriate answers and justifications. Following this adjustment, it approached ChatGPT 4.0 in accuracy and even outperformed it in terms of delivering responses that were comparable to those of human experts.
However, for some question kinds, both models were still inadequate. The algorithms performed well in analyzing securities transactions and tracking financial market trends, but they provided less accurate results in more specialized scenarios, such figuring out a client’s tax status and insurance coverage.
Together with Adam Bozman, a doctorate student at WSU, Fairhurst and Greene are now developing alternative methods to ascertain what ChatGPT can and cannot accomplish through a project that requests that it assess possible merger agreements. They are doing this by utilizing transactions done after September 2021, when the outcome is clear, and by exploiting the fact that ChatGPT is trained on data up until that point. According to preliminary results, the AI model isn’t particularly competent at this task yet.
The researchers concluded that ChatGPT is still likely to be more effective as a supporting tool than as a substitute for a qualified financial advisor. However, AI may alter how certain investment firms hire analysts at the entry level.
“The practice of bringing a bunch of people on as junior analysts, letting them compete and keeping the winners – that becomes a lot more costly,” said Fairhurst. “So it may mean a downturn in those types of jobs, but it’s not because ChatGPT is better than the analysts, it’s because we’ve been asking junior analysts to do tasks that are more menial.”
Also read: Viksit Workforce for a Viksit Bharat
Do Follow: The Mainstream formerly known as CIO News LinkedIn Account | The Mainstream formerly known as CIO News Facebook | The Mainstream formerly known as CIO News Youtube | The Mainstream formerly known as CIO News Twitter
About us:
The Mainstream formerly known as CIO News is the premier platform dedicated to delivering the latest news, updates, and insights from the CIO industry. As a trusted source in the technology and IT sector, we provide a comprehensive resource for executives and professionals seeking to stay informed and ahead of the curve. With a focus on cutting-edge developments and trends, The Mainstream formerly known as CIO News serves as your go-to destination for staying abreast of the rapidly evolving landscape of technology and IT. Founded in June 2020, The Mainstream formerly known as CIO News has rapidly evolved with ambitious growth plans to expand globally, targeting markets in the Middle East & Africa, ASEAN, USA, and the UK
