A recent study by The College Investor highlights worrying issues with Google’s AI-generated summaries for finance-related searches. According to their findings, many of these summaries contain inaccurate or misleading information, especially when it comes to more complex financial topics.
What the Study Found
Out of 100 personal finance searches, 43% of the AI-generated summaries were found to have incorrect or misleading details. The key findings include:
- 57% of the summaries were accurate.
- 43% had misleading or wrong information.
- 12% were completely incorrect.
- 31% were either misleading or missed important details.
Where AI Struggles
The AI was found to struggle the most with complex financial topics such as taxes, investing, and student loans. Some of the most concerning errors included:
- Outdated information on student loan repayment plans.
- Wrong details about IRA contribution limits.
- Misleading statements about 529 college savings plans.
- Incorrect tax information that could lead to penalties if followed.
While the AI handled simpler financial topics well, like basic definitions and general questions, it failed to account for important exceptions and recent changes in policies.
What AI Gets Right vs. Wrong
Here’s a breakdown of the types of queries Google’s AI tends to get right and wrong.
Queries AI Got Right
- Basic questions, like “What is a wire transfer?” or “How does a credit card work?”
- Simple, direct questions, like “Do I have to pay back student loans?”
- Recent trends, such as “What was the Chase glitch?”
- General insurance questions, like “When should I get life insurance?”
Queries AI Got Wrong
- Complex tax topics, such as “Can you use a 529 plan for a Roth IRA?”
- Nuanced financial products, like comparing an IUL to a 401k.
- Outdated student loan information or savings account rates.
- State-specific rules, such as California’s 529 plan details.
- Questions with context-dependent answers, like “Can I file as independent for FAFSA?”
Why This Matters
Google’s AI does well with simple questions, but it struggles with more detailed and complex issues. This means that while it’s a good tool for basic financial literacy, it’s not reliable for making important financial decisions. Relying on inaccurate information from AI could lead to financial mistakes, tax penalties, or bad investment decisions.
Robert Farrington, founder of The College Investor, expressed his concern: “If Google continues to present bad or misinformation about money topics to searchers, not only could it hurt their personal finances, but it could weaken already poor financial literacy in the United States.”
The study suggests that Google should stop using AI summaries for finance-related searches, especially those related to taxes and investments, to prevent potential harm to consumers.
Moving Forward
People searching for financial advice need to be cautious when relying on AI-generated summaries. While Google claims most AI overviews provide quality information, this study shows that errors can still occur, especially with complex or changing topics.
For those who want more details, The College Investor has published the full study on their website, complete with examples and methodology.