How AI May Be Undermining Your Investments

For the second time this year, Deloitte reportedly used generative artificial intelligence to “find” research citations that would help support their contracted reports to governments. The story is ultimately larger and has a lesson for how people manage their investment portfolios.

Deloitte And The Newfoundland And Labrador Report

Canadian blogger Matt Barter seems to have broken the first part of the most recent story. Using an access-to-information request (I gather it is like a freedom of information act in the U.S.), he found that the Government of Newfoundland and Labrador’s Department of Health and Community Services paid Deloitte Management Services $1,598,485 for a “Health Human Resource” plan that was to be a human resource strategy for the province’s healthcare sector.

Pricey, but not unusual. Governments frequently hire management consultant groups for analyses and reports. However, then came the other shoe drop. The Independent reported that the report contained at least four citations that didn’t appear to exist.

“The research papers cited in the 526-page document are used to support claims related to recruitment strategies, monetary recruitment and retention incentives, virtual care, and impacts of the COVID-19 pandemic on healthcare workers.” The Independent wrote.

One of the citations supposedly reported that monetary and recruitment incentives can save money. Martha MacLeod, a professor emerita in the University of Northern British Columbia’s School of Nursing, and a listed co-author of the citation, told The Independent in an email, “Our team certainly has done rural and remote nursing research. But we never did do a cost-effectiveness analysis, nor did we ever have the financial data to do it.” She further said that the citation was “false” and “potentially AI-generated.”

“Deloitte Canada firmly stands behind the recommendations put forward in our report,” a statement from the company said. “We are revising the report to make a small number of citation corrections, which do not impact the report findings. AI was not used to write the report; it was selectively used to support a small number of research citations.”

Not The First Time

This wasn’t a first. Critics raised in September AI usage concerns about a 10-year education plan for the same province, although not associated with Deloitte. According to the CBC, it included “at least 15 citations for non-existent journal articles and documents.” One of the references was to a 2008 movie that didn’t exist, but that did exist in a University of Victoria style guide that used it as an example of a fake reference.

Although Deloitte wasn’t connected to the education report, there was a story out of Australia. As Fortune reported, the company’s Australian member firm was paying a partial refund on a $290,000 report with alleged AI-generated errors. That included citations of non-existent academic research and “a fabricated quote from a federal court judgment.”

Such imaginary citations are examples of a phenomenon called large language model hallucination. The type of generative AI software doesn’t think or conduct research. In response to requests, it generates strings of text, based on immense samples of incorporated examples of writing, in response to requests. The technology is impressive. However, it is based on statistical use of words, not verified meanings, and while many find it helpful, there are circumstances in which it can’t be trusted because it doesn’t know what it’s doing.

Large language models, or LLMs, have received good and bad attention for various reasons. The issue of hallucinations, which can’t be programmed out of the software, is important. Pointing to research shouldn’t solely be an attempt to shape opinions. It is to help guide, challenge, and improve thought, strategy, and actions.

The Problem For Investors

Put differently, if someone includes a citation in a strategic document but hasn’t checked the source and read the material, they are interested in selling an idea, not in its ultimate quality. The problem isn’t limited to one global consulting firm or a particular part of Canada or Australia.

The practice is becoming more widespread. Damien Charlotin, a data consultant, has been tracking legal cases where lawyers used AI systems without sufficient oversight, resulting in 615 examples to date, including fabricated or misrepresented case law or incorrect language quoted from case law.

A study from researchers at the University of Texas at San Antonio, the University of Oklahoma, and Virginia Tech found a problem with AI code generators for Python and JavaScript programming languages can create new problems. Programmers frequently use software libraries with ready-to-use capabilities to speed development. Also, 97% of developers use gen AI to some degree, with 30% of code currently being generated by AI. Using 16 popular LLMs to generate code, the researchers found that the average percentage of library packages that are hallucinated and don’t exist is “at least 5.2% for commercial models and 21.7% for open-source models.” That means the code likely won’t work.

The problems are expanding and extending, which is why investors need to follow what their investment companies are doing. How many are sinking into some utter mess that may or may not show up explicitly in earnings announcements? How many are using consulting groups that aren’t checking carefully? What are the potential impacts on the companies and their operations?

Investors should start digging deeper during ongoing research to know the types of risks an investment is taking.