AI/ML and sensitive production data in fintech and healthcare? Where is the data going? Can it be made sense of? [D]

Hi all! i am currently working as a software engineer for a pretty big enterprise fintech company here in the states. In the the last 12 months at my job there has been a huge push for developers to use ai and agentic program in our development cycle, first in our ide directly, then Coder space instances with cloud agents and now code vulnerability remediation. This has gotten me thinking beyond developer productivity and more about how ML/AI systems can actually be integrated into production environments in highly regulated industries like fintech and healthcare. This makes me think "But hmmm... with a direct connection to sensitive production data, how do you design the architecture so that sensitive financial data doesn’t unnecessarily leave your environment? And if it does have to leave, how are companies handling PII??

I’m asking because a little bit of PII slipping into the cloud here and there might not seem like the end of the world, but imagine that integration has been running for a year or two. At that point, is that data potentially minable? like if there were ever a data leak at one of the AI provider companies, could that historical data potentially be analyzed or mined?