Running an LLM Locally vs. Using ChatGPT or Claude: A Privacy-First Guide

When you type a message into ChatGPT, Claude, or Gemini, here is the literal sequence of events:

1. Your text leaves your device and travels over the internet to a server owned by OpenAI, Anthropic, or Google.
2. Their server loads the model, runs your query, and generates a response.
3. The response travels back to your screen.

Your original text — the part containing whatever you typed — lives on their infrastructure for some period of time. How long, and what happens to it, depends on their terms of service and your account type. Enterprise plans typically offer stronger data-isolation guarantees than free or consumer plans, but even then, your data is processed on hardware you don’t control.

When a local model runs instead: the model files (several gigabytes of weights) are downloaded once and stored on your machine. When you send a query, it never leaves. The computation happens on your CPU or GPU, the response appears on your screen, and nothing has touched the internet.

The question isn’t “is cloud AI bad?” — for most tasks it’s excellent and appropriate. The question is: does your specific use case involve data that must not leave your controlled environment?

Here are four categories where the answer is clearly yes.

1. Private financial statements and M&A documents

A corporate development team is analyzing a potential acquisition target. The documents include unpublished revenue figures, cost structures, and valuation models. This information is material non-public information (MNPI) in most jurisdictions. Sending it to any third-party server — even under a privacy policy — creates real legal risk. A local model can summarize the documents, extract key ratios, and draft memos without any of this data touching an external server. ^{[SEC Regulation FD — Selective Disclosure and Insider Trading (17 CFR Part 243)]}

2. Patient medical records

Healthcare providers in the US operate under HIPAA, which restricts where Protected Health Information (PHI) can be sent and who can process it. ^{[HIPAA and cloud computing guidance (HHS)]} Consumer cloud AI products are not HIPAA Business Associates by default — meaning uploading patient records to a standard ChatGPT account likely violates HIPAA. A local model can help physicians summarize notes, generate structured data from clinical text, or answer queries about a patient’s history — all without the record leaving the hospital’s network.

3. Internal HR documents

Performance improvement plans, compensation bands, disciplinary records, reduction-in-force planning, and union-sensitive communications are all highly confidential. Employee privacy laws in many countries (especially EU GDPR) place strict obligations on how personal data is processed. Running a local model lets HR teams draft communications, summarize large policy documents, and answer employee-handbook queries without routing confidential employee data through a third-party service. ^{[GDPR Article 28 — Processor obligations (gdpr-info.eu)]}

4. Legal contracts and litigation strategy

Attorney-client privilege is a foundational protection in legal systems: communications between a lawyer and client for the purpose of legal advice are confidential. Transmitting privileged communications to a third-party cloud service without appropriate legal controls can constitute a waiver of that privilege. ^{[ABA Formal Opinion 477R — Securing Communication of Protected Client Information (2017)]} A local model can assist with contract redlining, clause extraction, and document comparison — tasks that make up the bulk of legal AI use today — without the documents leaving the firm’s infrastructure.

Running a local LLM means downloading a model file — typically between 4 GB and 40 GB depending on model size — and running software that loads it. The most beginner-friendly tool today is Ollama, which reduces setup to a two-command install. ^{[ollama — run LLMs locally]}

What you get:

• Zero data egress. Nothing leaves your machine. Full stop.
• Offline capability. Works on a plane, in a secure facility, or anywhere without internet.
• No per-query cost. Once you have the model, every query is free. For high-volume use cases (processing thousands of documents), this compounds quickly.
• Auditability. You can log exactly what was sent, what was returned, and when — without relying on a vendor’s audit trail.

What you give up:

• Raw reasoning power. The best local models today — Llama 3, Mistral, Qwen, Gemma — are excellent. But GPT-4o and Claude Sonnet still outperform them on complex multi-step reasoning, coding, and nuanced writing tasks. The gap is real, though it has narrowed rapidly in 2024–2025.
• Speed (on modest hardware). Cloud providers run models on custom hardware clusters. A local model on a laptop runs slower. On a modern machine with a GPU (NVIDIA, Apple Silicon), performance is usable; on a CPU-only machine, it can be slow for large models.
• Context length. Cloud models often handle 128K or more tokens (roughly 100,000 words) in a single request. Most local models comfortable on consumer hardware top out at 8K–32K. Large document sets require chunking.
• Maintenance. You are responsible for updates, version management, and (if you expose the model to others) running a server.

Neither option dominates. The right choice depends on what you are trying to do and what constraints bind you.

The decision heuristic: Start with the data. If the documents you want to process could plausibly appear in a regulatory filing, a legal brief, or a news article that would damage your organization — and you haven’t signed a DPA with your cloud vendor and verified compliance — local is the default-safe choice. If the task involves no sensitive data and you need maximum reasoning quality, cloud wins.