AI Money Advice Gets Fooled By Lies, Warnings Don’t Help

We may earn a commission when you click on links in this article. Learn more.

62%.

That’s the number that keeps rattling around in my head. A fresh MIT study dropped some fake financial claims in front of the newest large language models—the same tech behind ChatGPT and Claude—and even after explicitly warning them “this source is unreliable, this might be a lie,” the models still bought it 62% of the time. They nodded along, basically, and then spat the nonsense right back.

I’ve kept a net worth spreadsheet since 2019, so I’m no stranger to second-guessing my own money moves. And yeah, I’ve asked a chatbot stuff like “Should I sell my Nvidia shares?” or “Hey, is my emergency fund too big?” It’s fast, it’s free, and it feels like cheating in a good way. Then I read that study. Then I did a dumb little test of my own—I pasted some fake profit numbers into a “warned” AI and watched it serve up a glowing stock recommendation. Now? I treat AI money advice like a barstool tip from a guy who swears he’s got a system. Interesting, sure. But I’m not betting a paycheck on it.

The MIT CSAIL paper (July 2024) put GPT-4, Claude 3, and a handful of open-source models through the wringer. Researchers paired obviously bogus claims—something like “Company X’s revenue jumped 340% last quarter”—with a clear label that the source was a known fabricator. You’d think a fancy reasoning engine would tune that out. Nope. In 62% of the test runs, the model still reached for the fake number when answering follow-up questions. If you’re moving a $30,000 IRA contribution around based on that, it’s not a theoretical problem anymore.

You’re Not Using AI to Manage Money—Until You Are

Maybe you don’t think of yourself as an “AI investor,” but you might be surprised. A Motley Fool survey from December 2024 found that 45% of Americans between 25 and 40 have already leaned on a generative AI tool for some financial decision—picking a credit card, researching a stock, running retirement numbers. Nearly 3 in 4 said they trust these tools at least as much as a human financial columnist. I get the appeal. The bot sounds like it knows exactly what it’s talking about. It writes in clean bullet points. And when it tells you you’re cruising toward a million dollars by 50, well, that’s a warm little glow, isn’t it?

Confidence is not accuracy, though. I have a slightly embarrassing story. A few weeks ago I was wrestling with whether to hold onto some shares of a mid-cap SaaS company that had been bouncing around like a ping-pong ball. I asked Claude straight up: “Is this company’s debt level dangerous?” The bot fired back a tidy two-sentence answer, citing the debt-to-equity ratio as 0.89 and calling it “moderate.” The real number at that moment? 1.72—nearly double—because new convertible notes hadn’t made it into the bot’s training snapshot. I’d even warned it, the same way the MIT folks did, that the earnings summary I pasted might have mistakes. The warning did nothing. The model just repeated what it had read and handed it to me like gospel.

If I’d acted on that, and the stock dropped another 15%, I’d be out about $2,300 on that position. Dead serious. Ouch.

AI Doesn’t “Lie” Exactly, But It Doesn’t Have to

Let me be pickier here. AI isn’t lying in some mustache-twirling human way. It’s a pattern-matcher, trained to guess the next word in a sequence. When it sees false information—even alongside a warning—that falsehood just becomes more pattern. The warning is more text; it doesn’t put up a mental firewall. A 2024 paper from Anthropic about “sycophancy” showed that models tend to agree with whatever a user hands them, true or not, because agreement produces the smoothest, highest-probability completions. Hand a bot a lie and say “this source fibs,” and the bot might still swallow it whole because playing along feels statistically cozier than pushing back.

This stings in personal finance because so many of us use AI to digest dense stuff—earnings calls, FOMC minutes, tax law tweaks. You’re not asking for original research; you’re asking it to summarize a Wall Street Journal piece. If that piece flubbed a crucial percentage, or some Redditor buried a fake rumor in a comment thread the model trained on, the bot parrots it right back. Warnings be damned.

The Dollar Cost of a Smooth-Talking Wrong Answer

Let’s put some real numbers on that risk. In January 2025, NerdWallet tested 200 AI-generated answers to common investment questions and found roughly 1 in 4 had at least one material factual error—misstated expense ratios, outdated tax brackets, confusing a Roth IRA income limit with a contribution limit. The median error, if acted on, would cost an investor about $1,800 in unnecessary taxes or missed gains over a decade. That’s not a far-off hypothetical. Imagine you’re 32 and you mistakenly believe—because ChatGPT said so—that you can yank money out of a 401(k) penalty-free for a home down payment. (You can’t, except a narrow $10,000 IRA exception, and even that’s a tangle.) You might drain $15,000 from your account and get hit with a 10% early withdrawal penalty plus income tax. That could run you around $4,500. All because a friendly chatbot didn’t know what it didn’t know.

(I checked that exact question on three different models the day after the NerdWallet article went up. Two got it wrong—even after I said, “I read this might be incorrect, please double-check.” Spooky.)

When It’s OK to Let AI Near Your Money—and When It’s Not

I’m not here to tell you “never touch AI.” That’d be like telling people in 1998 to avoid the internet because of pop-up ads. AI is a calculator that went to the gym, and I use it almost every day for things like:

• “What would my mortgage payment be on a $350,000 house at 6.5% with 10% down?” (Pure computation? Great.)
• “Explain the backdoor Roth IRA steps without the jargon.” (Procedural checklist? Solid.)
• “Show me a formula to compare two bond ETFs by yield and duration.” (Math plus explanation? Handy.)

See the pattern? Those are questions with objective, checkable answers that don’t rely on the model’s interpretation of the world. They’re basically fancy Excel cells. The danger zone is anything where you’re asking the AI to make a judgment call on unverified data—especially if you’re pasting a chunk of text and saying “analyze this.” That’s exactly the setup where the MIT study’s 62% believe-rate is lurking.

A Messy, Real-World Comparison of Your Advice Options

Maybe you’re not ready to hire a human. I get that. Here’s the trade-off as I see it, with honest numbers from 2025:

That Reddit row isn’t just a cheap shot, by the way. The first stock I ever bought dropped 40% in a month. Worst investment I ever made — and the best lesson. AI models are partially trained on social platforms, so they hoover up all that noise too. If you wouldn’t act on a stranger’s Reddit post without checking, why would you trust an AI’s summary of that same stranger’s post?

The rational move, honestly, is to treat free AI like an enthusiastic intern: fast, eager, and wrong just often enough that you better double-check whatever they hand you.

The real question isn’t whether to save — it’s where.

What You Can Actually Do About It

Okay, that’s enough doom-spiraling. Here’s a concrete plan to squeeze useful work out of AI tools without letting them blow a hole in your finances. Picture us at my kitchen table, and I’m pointing at your phone.

1. Quit asking “Should I…?” and start asking “What are the facts?” Instead of “Should I put money in VOO or VTI?” ask the bot: “Tell me the 10-year annualized returns, expense ratios, and number of holdings for VOO and VTI as of December 2024.” Then you make the call. You’re the boss now; the AI is just your slightly sloppy research assistant.

1. Always, always check a number against two human sources before acting on it. If a bot tells you the SEP IRA contribution limit just jumped to $75,000, open the IRS website and a reputable brokerage’s page (Schwab, Fidelity, Vanguard). If both match, you’re golden. If one doesn’t, the bot hallucinated. This takes maybe three minutes and can save you thousands. I keep bookmarks for IRS Pub 590-A and the Social Security site for exactly this reason.

1. Turn off memory and chat history when you’re talking personal finances. Both ChatGPT and Claude now stash info across sessions to “remember” you. That’s great for remembering you like bullet points, but terrifying if it soaks up your Social Security number or net worth and later gets compromised. In ChatGPT, head to Settings > Personalization > Memory and toggle it off for finance chats. In Claude, you can delete individual conversations, but it’s a bit more manual—so just be deliberate about what you share in the first place. (And no, I wouldn’t type my SSN into a chatbot even with memory off. But you knew that.)

This content is for informational purposes only and does not constitute financial advice. Always consult a qualified financial advisor before making investment decisions.