Daniel Abraham, a 28-year-old software engineer, started getting messages from strangers on WhatsApp. It was not a data breach or a hacked website. It was Gemini, Google's AI assistant, handing out his personal phone number as if it were a company's customer-support line. Abraham flagged it to Google on March 17. The reply came on May 4.
His case is not an isolated accident, it is a symptom. These assistants do not go and look the information up at the moment they answer. They were trained by swallowing enormous amounts of text scraped from the internet, and inside that text sits real data about real people. The model memorizes it and, when it happens, spits it back to whoever asks. The underlying problem is this: you can delete a row from a database, but you cannot reach inside a model and pull your number out. Your life went in, and the machine does not know how to forget it.
It is not just the number
A University of Washington PhD student, Meira Gilbert, asked Gemini for the contact details of a collaborator, Yael Eiger, and the assistant handed her the personal phone number of a friend of Eiger's. "It was shocking," she said. And the damage does not stop at contacts. When Norwegian citizen Arve Hjalmar Holmen asked ChatGPT what it knew about him, he got back an invented horror story, that he had murdered two of his children and tried to kill the third, and was serving 21 years in prison, all mixed with true facts about his life, the number and gender of his children and the name of his home town. In March 2025 the privacy group noyb filed a complaint for breaching the GDPR, Europe's data protection regulation, because the system produced false information about an identifiable person. The AI does not only let real data slip out. It also attaches serious lies to real names.
Why the machine does not forget
Researchers have known for years that this is not a one-off glitch. A team led by researcher Nicholas Carlini managed to extract from an older model, GPT-2, whole sequences of memorized text containing real names, phone numbers and email addresses. They later showed that memorization grows with the size of the model in a predictable, almost proportional way, where making the model ten times bigger raises what it memorizes by about 19 percentage points. In plain terms, the more capable the assistant everyone is rushing to adopt, the more pieces of your life it can repeat.
And here is the knot the law has not untied. The GDPR gives you the right to be forgotten, the right to demand that your data be deleted. But that right was written for records that can be deleted, and a fact memorized by a model is not in a drawer, it is spread across billions of parameters. In Portugal, the data protection authority, the CNPD, has already put its finger on the wound. João Osório, of the CNPD, asked out loud: "How do we guarantee the right to be forgotten in generative AI, if the data has already been used to train the models?" The authority even proposed excluding certain names and profiles from AI-generated results. It is the official admission that the technology has outrun the law.
How to protect yourself
You cannot wipe the inside of a model, but you can shrink your exposure and know your rights. What helps:
- Reduce your footprint with data brokers. Much of what the AI memorized came from companies that buy and sell personal data. Ask to be removed from the ones that list you, and use search engines' "results about you" tools.
- Exercise your rights. The GDPR gives you the right to demand correction or deletion. If an assistant publishes something about you that is false or private, complain to the provider and, if needed, to your data protection authority.
- Do not feed the machine. Do not type personal data, yours or other people's, into chatbots, because what goes in can be retained.
- Distrust what the AI "knows" about people. A name paired with a claim is not proof. Treat an assistant's answer about someone as a rumour, not a record.
We got used to being able to delete a photo, close an account, change a number. With these models the logic has flipped, because your data may be inside, within reach of whoever asks, and getting it out stopped being a click you make and became an unsolved problem for whoever built the machine. Until they solve it, the memory is theirs and the nuisance is yours.
Sources: MIT Technology Review, noyb.
#StaySafe
🙏🖖
