Local LLM
A local LLM is a large language model that runs entirely on your own hardware — a laptop, workstation, or self-hosted server — without sending data to external API providers.
Understanding Local LLM
Cloud-based LLMs (GPT-4, Claude, Gemini) process your prompts on external infrastructure. Every query you send includes your data — email content, task descriptions, document text — which travels to and is processed by the provider's servers. For sensitive data, this creates privacy and compliance concerns.
Local LLMs eliminate this data exposure. Models like Llama 3, Mistral, Gemma, and Phi run entirely on your own hardware using tools like Ollama, LM Studio, or llama.cpp. Your data never leaves your machine. The tradeoff is capability and speed: local models are generally less capable than frontier cloud models, and running large models requires significant GPU hardware.
The gap between local and cloud LLMs is narrowing rapidly. Llama 3 70B approaches GPT-4 quality on many tasks. Quantization techniques reduce model sizes dramatically — a 70B model can run on consumer hardware when quantized to 4-bit precision. For specific domains and tasks (especially those requiring privacy), local LLMs are increasingly viable.
Hybrid approaches are emerging: use a local LLM for sensitive, personal data processing, and a cloud LLM for tasks requiring maximum capability where the data is less sensitive.
How GAIA Uses Local LLM
GAIA does not run local LLMs and cannot be pointed at a local model server such as Ollama or LM Studio. It runs on hosted models from supported providers, and a self-hosted install also needs an API key for one of them (OpenAI or Google Gemini). Self-hosting with Docker keeps GAIA's databases and services, including your email, task, and calendar data, on your own infrastructure, but each model request is still sent to the provider you choose. If everything must stay on your machine, use a local tool such as Ollama or LM Studio directly.
Related Concepts
Self-Hosting
Self-hosting is the practice of running software on your own servers or infrastructure instead of using a cloud-hosted service, giving you complete control over your data, configuration, and availability.
Data Sovereignty
Data sovereignty is the principle that data is subject to the laws and governance of the jurisdiction where it is stored, and that individuals and organizations have the right to control where their data resides and who has access to it.
Large Language Model (LLM)
A Large Language Model (LLM) is an artificial intelligence model trained on vast amounts of text data that can understand, generate, and reason about human language with remarkable fluency.
Open Source AI
Source-available AI refers to artificial intelligence software whose source code is publicly available, allowing anyone to inspect, modify, distribute, and contribute to the project.
Foundation Model
A foundation model is a large AI model trained on broad data at scale that can be adapted to a wide range of downstream tasks through fine-tuning, prompting, or integration into application architectures.
