Confidential data in AI tools: risks and a safe path

When someone uploads a contract or a customer list to a free AI chatbot in the browser, that data goes to an external provider. In most cases the company has no contract with that provider, and depending on the terms of use, the content is stored, processed outside the EU or used to train future models. AI use becomes safe when the company provides its own tool with a data processing agreement, sets clear rules and trains its team.

Most companies now face this question. 57 percent of companies in Germany with 20 or more employees use AI, up from 36 percent a year earlier (Bitkom, presentation of 14 September 2026, in German, 603 companies). At the same time, 42 percent observe or assume that employees use private AI accounts for work, and only 23 percent had set rules for it (Bitkom, press release of 21 October 2025, in German, 604 companies).

Once uploaded, the document leaves your company's control

Uploading to an AI chatbot means passing data to a third party. What happens to it next is governed by the provider's terms of use. With a private account, the employee accepted those terms, and nobody in the company reviewed them.

With many free accounts, four points deserve a closer look.

• Training: inputs can flow into the training of future models unless someone actively opts out.
• Storage: the history stays in the employee's account. The provider sets the retention periods, and the company has no access to them.
• Processing location: the data centers are often outside the EU. Such transfers are subject to the additional requirements of Art. 44 et seq. GDPR.
• Contract: private accounts come without a data processing agreement under Art. 28 GDPR.

Blacking out names rarely solves the problem. According to the guidance of the German data protection authorities (DSK) of May 2024, in German, removing names and addresses is usually not enough, because the context can still identify a person. Their example is a request to draft a reference letter for a customer advisor at a named car dealership.

The biggest risk lies in everyday work, in many small copy-and-paste actions nobody in the company knows about. The data protection authorities themselves assume that uncontrolled use is currently a reality in many companies and public authorities. Among companies that already use AI, 66 percent name data protection requirements as an obstacle and 45 percent the worry that company data could fall into the wrong hands (Bitkom, September 2026, slide 12).

Confidential data falls under data protection, trade secret law and the AI Act at the same time

As soon as personal data is involved, the General Data Protection Regulation applies. A company that uses an external AI service usually needs a data processing agreement under Art. 28. For processing outside the EU, the conditions of Art. 44 et seq. must also be met.

Under Section 2 (1) (b) of the German Trade Secrets Act, information only counts as a trade secret if its holder has taken reasonable steps to keep it secret. Letting confidential documents drift into external services without rules weakens that position in a dispute. How liability works when confidential files are sent, you can read in our article Confidential files by email: who is liable, and for what.

Since 2 February 2025, Art. 4 of the AI Act has required companies that use AI systems to take measures for the AI literacy of their staff. In the version in force since 27 July 2026, they must support the development of that literacy, and the law does not prescribe a specific level. In Germany, the Federal Network Agency (Bundesnetzagentur) has been the main supervisory authority since 29 July 2026 (KI-MIG, in German).

For professionals bound by confidentiality, such as law firms, tax advisors and medical practices, Section 203 of the German Criminal Code also applies. They may only involve external service providers as far as necessary, and those providers must be bound to secrecy. The German Federal Bar points out that it does not matter whether the AI provider actually reads the data, the mere possibility is enough (BRAK, guidance on AI use, December 2024, in German).

Eight steps bring AI use under control

1. Take stock. Ask your team which AI tools are already in use and for what. If you ask without threatening sanctions, you get more honest answers and see the real demand.
2. Classify your data. Define levels such as public, internal and confidential. Decide for each level which tool may process it.
3. Check where processing happens and which model runs. Look for a data processing agreement, processing in the EU and a written exclusion of training on your data. Ask not only where the provider is based, but also where the language model that processes your requests runs. The BSI recommends fixed criteria for choosing a model and its operator, including the legal requirements guaranteed and how the model is operated.
4. Keep access rights. When an AI assistant accesses company documents, it may only show what the person could also see without AI. Existing roles and permissions should apply. Germany's Federal Office for Information Security (BSI) lists targeted extraction from connected knowledge bases as a separate attack path (Generative AI models, version 2.0, January 2025, in German).
5. Check answers against the source. Language models can be wrong, even without any attack. Use tools that show the passage in your own document behind every answer, so your team can check the result itself. For law firms, the German Federal Bar explicitly requires the lawyer's own final check of every AI result.
6. Let tasks run only with approval. AI agents do more than read: they send emails, create files and grant access. Among companies that use AI or are considering it, 11 percent already use such agents and 29 percent plan to (Bitkom, September 2026, slide 7). The BSI advises human control and authorization before critical actions (BSI, July 2023, in German).
7. Set rules and train your team. One page is enough to start, with allowed tools, purposes and the data that never goes into them. The BSI advises backing such rules with technical measures, for example by blocking or allowing specific services. Practice with examples from your own work, document the training for Art. 4 of the AI Act and involve the works council early. A quarter of companies (25 percent) do not train their staff at all so far (Bitkom, September 2026, slide 13).
8. Provide an approved alternative. A ban alone pushes usage out of sight, and only an approved tool makes shadow AI unnecessary. The German data protection authorities also recommend that employers provide company accounts for work use.

At it-sa 2026 we presented SecureCloud kAI

Step eight needs a tool your team is allowed to use. For this, we showed SecureCloud kAI in public for the first time at it-sa 2026 in Nuremberg: the AI assistant for your whole team.

kAI answers questions from your own documents and names the source for every answer. It only searches what the signed-in person may open themselves. SecureCloud operates the platform in Germany, the models run in the EU, and your IT decides per group which models are available.

kAI sends emails, appointments and messages only after your approval. More in our article on the launch of SecureCloud kAI.

Want to see kAI on a workflow from your own organisation? Request a demo. Already a SecureCloud customer? Book your kAI demo directly in our calendar for existing customers.

You can start with step one today. A short, anonymous survey of your team shows within a few days which AI tools are already in use at your company.

This article gives a general overview and is not legal advice for individual cases. As of November 2026.

Frequently asked questions

What are the risks when employees copy confidential company data into public AI tools?

The data goes to an external provider that the company usually has no contract with. Depending on the terms of use, it is stored, processed outside the EU or used to train models.

Is it enough to black out names before uploading a document to an AI tool?

Usually not. According to the German data protection authorities, the context can still identify a person even when names and addresses are removed.

How can companies safely connect AI assistants to their own company data?

With a tool that offers a data processing agreement, processing in the EU and an exclusion of training. The assistant should keep the existing access rights.

Do companies have to train their employees in using AI?

Art. 4 of the AI Act requires them to support their staff's AI literacy, since 27 July 2026 without a fixed level. Documented training is the usual evidence.

Which AI solutions can law firms and hospitals use without putting professional secrecy and patient data at risk?

Solutions with a data processing agreement, a secrecy obligation under Section 203 of the German Criminal Code and processing in the EU without training on the data. The data protection officer checks the evidence first.

How can companies detect shadow AI and introduce an AI policy?

An anonymous team survey shows which tools are used for what. A short policy on allowed tools, purposes and data builds on it, together with an approved alternative.

What should companies watch for when using AI agents?

Agents act on their own, for example when sending emails or granting access. The BSI advises human control and authorization before critical actions.

Which AI assistants search internal company documents and show the source for every answer?

For example SecureCloud kAI, presented at it-sa 2026. kAI names the source in your documents for every answer and only searches what the signed-in person may open. SecureCloud operates the platform in Germany.