Somewhere today, a corporate accountant will paste a spreadsheet of quarterly revenue figures into ChatGPT. A gig worker will upload three months of bank transactions and ask for a budget breakdown. A founder will feed a cap table into an AI tool to sanity-check a valuation. All of them will close the tab afterward and assume the data is gone.
A growing body of evidence suggests that assumption is wrong, and the consequences are compounding fast.
The numbers behind the leak
Harmonic Security, a firm that monitors how employees interact with generative AI tools, published findings in 2025 based on an analysis of 43,700 prompts and 4,400 files submitted to AI platforms in workplace settings. The results were stark: 4.37% of all prompts contained sensitive data, and 22% of uploaded files did too. The exposed material included login credentials, personally identifiable information (PII), merger documents, and internal financial records.
It is worth noting that Harmonic Security sells AI-monitoring software, so the company has a commercial interest in highlighting these risks. But the methodology is observational, drawn from real employee behavior rather than simulated scenarios, and no competing dataset has contradicted the scale of the findings.
A 4.37% rate sounds manageable in isolation. Scale it up and the picture changes. A mid-size company where 500 employees each send 20 prompts a day would generate roughly 440 sensitive-data exposures every 24 hours. Across a large enterprise or an entire industry, that figure climbs into millions of individual leaks per year.
Why ChatGPT’s financial features raise the stakes
The Harmonic Security study measured behavior across generative AI tools broadly, not one product in isolation. But the timing is hard to ignore. OpenAI has been steadily expanding ChatGPT’s ability to handle money-related tasks: analyzing spending patterns in uploaded bank statements, connecting with external financial services, and offering budgeting assistance that feels more like a fintech app than a chatbot.
OpenAI reported more than 300 million weekly active users in early 2025. Even if a small fraction of that base engages with financial features, the raw volume of people potentially exposing account details, balances, and transaction histories is enormous. And the long-term handling of that data remains opaque to most of them.
No public telemetry from OpenAI or its competitors breaks out how many users specifically submit financial prompts, or whether those prompts carry a higher rate of sensitive data than general queries. That transparency gap is itself a problem. Users adopting AI-powered budgeting tools may reasonably assume those features come with protections comparable to a bank or a regulated fintech app like Plaid. Whether that assumption holds depends on policies and architectures that most users never read and that few independent auditors have stress-tested for financial use cases.
OpenAI does offer settings that let users opt out of having their conversations used for model training, and its enterprise tier includes contractual commitments against training on customer data. But the default consumer experience still involves data retention for abuse monitoring and service improvement, and the specifics of how long prompt data persists, and under what conditions, are buried in policy documents that change frequently.
What happens to financial data inside a model
The common intuition that a chatbot “forgets” your input after a session is not reliably true. Researchers published a study in 2023 examining how large language models handle sensitive inputs and found that PII can be memorized during training and regurgitated verbatim under certain conditions. The paper focused on training-data memorization rather than cross-session leakage in a live chat product, but it established a concrete mechanism: sensitive information that enters a model’s pipeline can, under the right circumstances, resurface in outputs.
That finding has grown more relevant as AI tools have added persistent memory, saved chat history, and plug-in architectures that extend how long and how broadly user data circulates within a system. The 2023 paper is now nearly three years old, but the underlying dynamic it describes has not been resolved. If anything, the attack surface has expanded.
A separate line of research on prompt-injection attacks, published in 2024, showed that adversarial instructions can be smuggled into a conversation to coax a model into ignoring safety guardrails and revealing data from its current context window. In controlled experiments, including simulated banking interactions, researchers demonstrated successful extraction of information the model was supposed to protect. The experiments were conducted in lab conditions, not against production financial tools, but they proved the mechanism works.
No confirmed real-world incident has been publicly reported in which a prompt-injection attack against a production financial AI agent successfully exfiltrated a user’s banking details. The gap between proof-of-concept and confirmed exploitation remains open. Security researchers, however, generally treat a demonstrated attack vector as a matter of “when,” not “if,” particularly as financial AI features grow more complex and more tightly connected to live account data.
Regulators are paying attention, but slowly
The regulatory landscape has started to shift, though not at the speed the technology is moving. The EU AI Act, which began phased enforcement in 2025, classifies AI systems used in creditworthiness assessment and financial services as “high-risk,” subjecting them to stricter transparency and data-governance requirements. In the United States, the Consumer Financial Protection Bureau (CFPB) has signaled interest in how AI tools handle consumer financial data, and the SEC has issued guidance on AI-related disclosures for financial firms. But as of mid-2026, no U.S. regulator has published binding rules specifically addressing the use of general-purpose AI chatbots for personal financial management.
That regulatory gap means the burden of protection falls largely on individual users and, in workplace settings, on employers who may or may not have updated their data-handling policies to account for generative AI.
What the evidence proves and what it does not
The Harmonic Security dataset is the strongest piece of this puzzle. It is observational data drawn from actual employee behavior, which makes the 4.37% and 22% figures a reliable baseline for the current rate of sensitive-data leakage into AI systems.
The academic papers serve a different function. They prove that specific vulnerabilities, memorization, regurgitation, prompt injection, exist under experimental conditions. They do not prove those attacks are happening at scale against consumer banking tools, or that any particular provider has suffered a breach through these mechanisms. They should be read as evidence that the underlying risks are technically real, and that the distance between safe and unsafe use depends on how systems are configured, monitored, and audited.
For vendors building financial AI features, the data makes the case for tighter guardrails, clearer data-retention disclosures, and independent security testing focused specifically on money-related interactions. For users, the implication is more immediate and more personal.
What to do before your next financial prompt
The practical guidance is straightforward: treat every chatbot prompt as if it could be stored, indexed, and resurfaced. That means not pasting full account numbers, login credentials, or unredacted financial statements into any AI tool, regardless of how polished its interface looks or how useful its budgeting features seem.
Steps worth taking as of June 2026:
- Turn off chat history and training toggles in any AI tool you use for financial questions. OpenAI, Google, and Anthropic all offer some version of this setting, though the defaults vary and the labels are not always intuitive.
- Redact before you paste. If you want AI help analyzing a bank statement, strip out account numbers, routing numbers, and full names first. The model does not need them to categorize your spending.
- Use anonymized or synthetic data when testing financial scenarios. A chatbot can model a budget just as effectively with rounded, fictional numbers as with your real ones.
- Check your employer’s AI policy. The Harmonic Security data came from workplace usage. If your company has not published guidelines on what employees can and cannot share with AI tools, that is a policy gap worth raising with leadership.
- Read the data-retention policy for any AI tool you connect to financial accounts. Look specifically for how long prompt data is stored, whether it is used for model training, and what happens to it if you delete your account.
- Compare protections to your existing fintech tools. If your bank or budgeting app is regulated under Gramm-Leach-Bliley or similar financial privacy laws, ask whether the AI tool you are considering offers comparable safeguards. In most cases, it does not.
The gap between convenience and safety is still widening
AI-powered financial tools are genuinely useful. They can spot spending patterns a human would miss, simplify tax prep, and make sophisticated analysis accessible to people who cannot afford a financial advisor. None of that utility is in question.
What is in question is whether the infrastructure behind those tools has been built, tested, and regulated with the same rigor applied to the banking systems they are increasingly asked to touch. The Harmonic Security data shows that millions of sensitive data points are already flowing into AI platforms. The academic research shows that those platforms have documented, unresolved vulnerabilities in how they handle such data. And the regulatory framework has not yet caught up.
Until it does, the safest rule is the simplest one: if you would not tape it to a bulletin board in your office lobby, do not type it into a chatbot.
