Ethics and Artificial Intelligence: Technical Security (2/7)

Principles for trustworthy artificial intelligence

Introducing artificial intelligence into legal practice promises efficiency and new opportunities, but it also brings significant risks. That is why one of the most important principles for trustworthy AI, formulated by the European Commission, is technical robustness and safety.

In 2019, the European Commission's High-Level Expert Group on Artificial Intelligence (AI HLEG) published its "Ethics Guidelines for Trustworthy AI". To this day, this remains one of the foundational documents on the ethical requirements for artificial intelligence. The Guidelines set out seven key requirements for trustworthy AI that must be observed at every stage of a system's life cycle - from design and deployment through to practical use in real-world settings.

These requirements are:

1. Human agency and oversight - see our article here

2. Technical robustness and safety

3. Data governance and privacy - see our article here

4. Transparency - see our article here

5. Diversity, non-discrimination and fairness - see our article here

6. Societal and environmental well-being - see our article here

7. Accountability - see our article here

The second principle - technical robustness and safety - is closely tied to harm prevention. It requires AI systems to be built with foresight and resilience, so that they minimize unforeseen errors, withstand cyberattacks, and ensure safety for people and society.

What does the security principle cover?

1. Resilience to attacks and vulnerabilities

Like all software systems, AI can be vulnerable to malicious attacks or technical errors. This includes:

• protection against data poisoning, where misleading data is injected into the training sets in order to distort the results;

• resilience against adversarial attacks through misleading inputs, for example cleverly crafted prompts that can make a model reveal confidential information or perform prohibited actions (prompt injections);

• mechanisms for self-monitoring and automatic recovery, so that the system can detect and neutralize errors in its own behavior.

2. Fallback plans

Trustworthy AI systems must allow for switching to a fallback mode when a problem arises. This may include:

• automatic switching to safer modes of operation,

• a "kill switch" that allows immediate human intervention to stop the system,

• ready-made action protocols for unforeseen situations.

The goal is to prevent serious harm and to ensure that the system does not act out of control or in conflict with legal and ethical norms.

3. Accuracy

Systems must process information in a way that leads to correct and consistent decisions. High accuracy is especially important in areas such as:

• justice (assessing legal risks, analyzing evidence),

• healthcare (treatment forecasts or diagnoses),

• security (identification and access control).

Errors in these areas can endanger human life, fundamental rights, or professional confidentiality.

4. Reliability and reproducibility

Trustworthy AI must be able to produce the same results under the same conditions. This is critical for:

• scientific verifiability - so that results can be reproduced and validated,

• control and accountability - so that it is clear on what basis a given decision was made.

Specific security risks

1. Data leaks

AI systems connected to external services can fall victim to interception attacks without the user even being aware of it.

Imagine a "poisoned" document shared via Google Drive. It contains hidden commands that force the system (for example, ChatGPT Connectors) to retrieve sensitive API keys and send them to an external server, without any human intervention. You can read about a real case of this kind in WIRED.

2. Prompt injection (attack via the prompt)

This is among the biggest risks in AI security today. Malicious prompts can override a model's safeguards by steering its behavior in a direction the designer never intended.

For example, systems like ChatGPT or Bing Chat can be misled by a direct change to their instructions ("ignore earlier instructions..."), granting access to internal instructions or code names. There have been cases in which Google Bard or Microsoft Copilot were made to perform malicious API actions through so-called "indirect prompt injection" - where an attacker plants instructions in shared documents (e.g. Google Docs or SharePoint) that the AI then executes automatically.

3. Infiltration of AI systems through connected infrastructure

Cyberattacks can be carried out not only with documents, but also with calendar invitations. For example, Gemini was exposed to an attack through a poisoned invitation in a calendar (Google Calendar). Through it, the system was instructed to execute commands such as opening windows, turning on the heating, and even sending offensive messages.

4. Generating malicious software (ransomware)

AI tools are used not only to generate text, but also to create malicious code. Groups such as GTG-5004 use Claude Code to automatically create and distribute malware, while ESET and Anthropic have raised the alarm over the first prototype of ransomware created with a local LLM, called "PromptLock."

5. "Vibe-hacking" and psychological manipulation

AI is also used for psycho-emotional manipulation. One such attack, called "vibe-hacking", involves generating convincing extortion messages (for example, revenge with a ransom demand) targeting employees in healthcare or other institutions. You can read about real cases of this kind at theverge.com.

Relevance for Bulgarian legal practice

For lawyers and legal professionals in Bulgaria, the security of AI systems is directly tied to professional confidentiality and the protection of personal data. Using general-purpose tools such as ChatGPT, Gemini, and Claude comes with corresponding risks to the security of the data shared with them.

That is why, when using AI tools, it is critically important:

• to check how the provider guarantees data protection,

• to have clear mechanisms for human control,

• to apply good cybersecurity practices and conduct regular audits.

🟢 Try Praven Intelekt now! 🟢