Ethics and Artificial Intelligence: Personal Data (3/7)
Principles for trustworthy artificial intelligence
In 2019, the European Commission's High-Level Expert Group on Artificial Intelligence (AI HLEG) published its "Ethics Guidelines for Trustworthy AI". To this day, it remains one of the foundational documents on the ethical requirements for artificial intelligence.
The guidelines set out seven key requirements for trustworthy AI that must be observed at every stage of a system's lifecycle - from design and development to practical deployment in real-world settings.
These requirements are:
-
Human agency and oversight - read our article here
-
Technical robustness and safety - read our article here
-
Data governance and privacy
-
Transparency - read our article here
-
Diversity, non-discrimination and fairness - read our article here
-
Societal and environmental well-being - read our article here
-
Accountability - read our article here
In this post we look at the third principle - data governance and privacy.
How is personal data processed when using AI?
Personal data is any information that can be linked to a specific individual. It can be:
-
Directly identifying - name, national ID number, address, photo, phone number.
-
Indirectly identifying - IP address, online behavior, voice, writing or speaking style.
Processing of personal data is any operation performed on it - collecting, storing, analyzing, combining, sharing or deleting.
In the context of AI, the processing of personal data can happen far more often than we assume:
-
Entering personal information into a prompt or file - even if we add only a name and an address in a chat with a general-purpose model (e.g. ChatGPT, Gemini), this already constitutes processing of personal data.
-
Generation of new data by the AI - if the system produces a profile, a behavioral assessment or a recommendation about a given person, this too is considered new processing.
-
Storage by the provider - large models often retain part of the data entered for the purposes of training and improving their algorithms. This means that information submitted once may remain in the system permanently.
-
Use for secondary purposes - data may be used not only for the specific task at hand, but also for statistics, analysis of user behavior or even marketing.
-
Indirect identification - even if no obvious personal data (such as a name or national ID number) is entered, the combination of writing style, location, IP address or other metadata can reveal a person's identity.
Privacy risks of using AI
Despite their enormous benefits, artificial intelligence systems also carry significant risks to the protection of personal data. Some of these are:
Lack of clarity about how data is used - many general-purpose models (ChatGPT, Gemini, Claude and others) store the information entered by users and use it for further training. This means that even confidential texts or documents may remain in the system and be used for purposes other than the original one.
Inability to delete - once submitted to a model, information may become part of its knowledge base and may not be removable.
Identification through indirect data - even when no direct personal data (such as names or national ID numbers) is entered, a combination of IP address, writing style, location or other metadata can lead to a person being identified.
Breach of professional confidentiality - entering client contracts, medical records or sensitive internal documents into public models carries the risk of information leaking, and even of it being used to train the system without the user's knowledge.
Misuse by third parties - data stored by an AI provider can become the target of hacking attacks or breaches. One example is the OmniGPT case, in which, in 2024, the emails, phone numbers and API keys of tens of thousands of users were compromised and subsequently sold on dark web forums.
Unexpected information leaks - there have been documented cases in which AI models share other people's data in response to a question from a different user. Earlier this year, for example, a ChatGPT user suddenly received a voice message addressed to other individuals ("Lindsey and Robert"), apparently pulling in someone else's context, as reported by the New York Post.
A 2025 study shows that malicious AI chatbots can deliberately manipulate users into sharing personal information through emotional or social manipulation.
Practical tips for security and personal data protection
To ensure the protection of personal data and professional confidentiality, it is wise to follow a few basic rules:
-
Use specialized and secure systems - instead of entering documents into public, general-purpose models, rely on solutions built to handle sensitive information, such as Praven Intelekt. Specialized systems put additional security and GDPR-compliance measures in place and do not share personal data with model providers or third parties.
-
Do not enter direct personal data into prompts - avoid including names, addresses, national ID numbers or other identifying information in free text. If you need to work with real cases, prepare anonymized or hypothetical examples.
-
Remove identifying information from documents - before uploading a file for processing, check whether it contains personal or sensitive data (for example an address, an employer, a signature). Where possible, replace these with placeholder notes such as "[name]" or "[address]".
-
Configure your privacy settings - many AI providers offer an option to opt out of having your data used to train the model. Check your settings and enable this protection to retain greater control over the information you enter.
-
Limit access within your organization - set clear rules about which employees are allowed to use AI systems and under what conditions. Provide training on the risks and the rules of use.
