The Privacy Stakes in LLM Applications
Every LLM API call is a data transfer. The text sent in your prompts — user queries, document contents, conversation history — travels to a third-party server, is processed by a model, and generates a response. For most types of content this is unremarkable. For content that includes personal data, regulated information, confidential business information, or sensitive communications, it creates privacy and compliance obligations that many teams deploy LLM applications without fully understanding. The consequences of getting this wrong range from regulatory fines and data breach notifications to loss of customer trust and contractual liability. This guide covers the practical privacy and compliance considerations that every LLM application builder needs to understand before sending data to a model API.
What Data You Are Sending and to Whom
Start by mapping your data flows. For each LLM API call in your application, identify: what categories of data appear in the prompt (user-generated text, retrieved documents, conversation history, structured data from your database)? Does that data contain personal information — names, email addresses, phone numbers, IP addresses, location data, health information, financial data? Does it contain regulated information — PHI under HIPAA, financial data under GLBA, children’s data under COPPA, EU personal data under GDPR? Does it contain confidential business information that should not leave your organisation? This mapping is the starting point for understanding your privacy exposure. Most teams are surprised by how much personal data appears incidentally in prompts — a customer support conversation that includes an order ID can be linked to a specific person; a document analysis prompt that includes author metadata contains a name; a meeting transcription contains names throughout.
Data Processing Agreements and Terms of Service
Before using any LLM API in production with personal data, you need a Data Processing Agreement (DPA) in place with the provider — this is a legal requirement under GDPR and analogous requirements in other jurisdictions. A DPA specifies how the processor (the LLM provider) handles the personal data on your behalf, what security measures they maintain, what they do with the data after processing, and what happens in case of a data breach. Anthropic, OpenAI, Google Cloud, and AWS all offer DPAs for enterprise API users. These are typically available via your account settings or enterprise agreements. The standard consumer-facing terms of service for tools like Claude.ai or ChatGPT are not DPAs and are not sufficient for processing personal data on behalf of your users — you need the API with an enterprise agreement.
Read the data handling provisions in your provider’s terms carefully. Key questions: does the provider use your API inputs to train their models? If so, you may be inadvertently contributing your users’ personal data to training datasets without appropriate consent. OpenAI and Anthropic both offer API plans where inputs and outputs are not used for training — verify which plan you are on. Does the provider store your API inputs? For how long? In which geographic regions? These questions matter for data residency requirements and breach notification obligations.
GDPR Considerations for LLM Applications
If your application processes personal data of EU residents — which includes most B2C applications serving European users and many B2B applications whose users include European employees — GDPR applies regardless of where your organisation is headquartered. Several GDPR provisions have specific implications for LLM applications. Lawful basis: you need a lawful basis for processing personal data, and sending it to an LLM API is processing. Legitimate interest, contractual necessity, or explicit consent may apply depending on your use case — document which basis you rely on for each data category. Purpose limitation: personal data collected for one purpose cannot be used for an incompatible purpose. If users provide personal data to use your product, using that data to train models without separate consent may violate purpose limitation. Data subject rights: users have the right to access, rectify, and erase their personal data. If personal data has been included in LLM API calls, you need processes to handle erasure requests — which is technically complex when data has been sent to third-party APIs. Data transfers: transferring personal data to a LLM provider in a third country (e.g., US) requires appropriate safeguards — Standard Contractual Clauses are the most common mechanism for EU-US transfers.
Figure 1 — Data Sensitivity Tiers for LLM API Decisions
Sector-Specific Regulations
Healthcare (HIPAA). Protected health information (PHI) — any individually identifiable health information — cannot be processed by a Business Associate without a signed Business Associate Agreement (BAA). Most general LLM API providers are not set up to sign BAAs for standard API plans. Microsoft Azure OpenAI Service and AWS Bedrock both offer paths to HIPAA-compliant LLM deployment with appropriate BAAs. Alternatively, self-hosted open-source models keep PHI within your controlled infrastructure and eliminate the BAA question. Never send PHI to a general LLM API without confirming your provider has signed a BAA covering that API service.
Financial services. Financial data — account information, transaction details, credit decisions — is regulated under GLBA in the US, and financial services firms face additional regulatory expectations from their prudential regulators (OCC, FRB, FCA, FINMA) regarding AI systems used in regulated activities. Many financial services regulators require that firms be able to explain AI-assisted decisions, maintain audit trails, and demonstrate that AI tools do not introduce discriminatory bias into lending or underwriting decisions. Build explainability and audit logging into LLM-assisted financial workflows from the start.
Legal privilege. Attorney-client privileged communications cannot be disclosed to third parties without waiving privilege. Sending privileged legal communications to a cloud LLM API is disclosure to a third party and may waive privilege under applicable law, depending on jurisdiction and specific circumstances. Legal teams using LLMs for privileged work should use self-hosted models or obtain specific legal advice about whether their use of cloud APIs constitutes a privilege waiver.
Practical Mitigations: Reducing Privacy Exposure
Several practical techniques reduce privacy exposure without requiring self-hosting. PII scrubbing: strip personal identifiers from prompts before sending to the API — replace names with placeholders, mask email addresses, remove phone numbers. Libraries like Microsoft Presidio, spaCy NER, and cloud DLP services automate this. The limitation is that scrubbing is imperfect and some context may be lost. Minimisation: only include in the prompt the information actually needed to answer the query — do not send entire user profiles when only a specific attribute is relevant. Pseudonymisation: replace real identifiers with consistent pseudonyms within a session, allowing the model to reason about “User A” and “Company B” without knowing who they actually are. The mapping is retained server-side and can be used to re-identify the output. Synthetic data for development: use synthetic or anonymised datasets during development and testing so real personal data is never sent to APIs in a non-production context where additional scrutiny may be lower.
Building a Privacy-by-Design LLM Architecture
The most durable approach to LLM privacy compliance is designing privacy into the architecture from the start rather than retrofitting it later. This means: defining data sensitivity tiers for every data type in your application and the routing rules that direct each tier to the appropriate processing infrastructure; implementing DPA-compliant provider agreements before any personal data reaches production; building PII detection and minimisation into your data pipeline as a mandatory step before prompt construction; maintaining logs of what data was sent to which provider with timestamps, for incident response and data subject request fulfilment; and implementing a data retention policy for API inputs that aligns with your broader data retention obligations. Privacy-by-design is not a one-time project — it is an ongoing discipline that requires regular review as your data flows evolve, new providers are added, and regulations change. Building it into your engineering culture from the start is significantly easier than retrofitting it onto an established application.
Incident Response for Privacy Breaches Involving LLMs
If personal data sent to an LLM API is involved in a data breach — whether through provider-side compromise, misconfigured access controls, or inappropriate use of API outputs — you face the same notification obligations as any other data breach. GDPR requires notification to the supervisory authority within 72 hours of becoming aware of a qualifying breach, and notification to affected individuals without undue delay for breaches likely to result in high risk. CCPA, state breach notification laws, and HIPAA have analogous requirements with varying thresholds and timelines. Your LLM API data flows should be included in your breach response plan, with clear answers to: how would you know if data sent to the API had been compromised? What data was involved, how many individuals affected, how do you identify and notify them? Having these answers documented before an incident means you can focus on response rather than discovery during an actual breach. Privacy compliance in LLM systems is fundamentally the same discipline as privacy compliance in any data processing system — the specific techniques differ, but the underlying principles of minimisation, purpose limitation, security, and accountability apply directly and fully.
The teams that handle privacy well are those who treat it as an engineering discipline with the same rigour they apply to security, reliability, and quality — not as a compliance checkbox visited once at launch and never revisited.