What llama.cpp actually is and why you already use it
llama.cpp is a C++ library that loads quantized LLM model files (GGUF format) and runs inference on CPU, GPU, or hybrid. It has zero Python dependencies, compiles with a single make command, and the entire binary is a few megabytes. Ollama wraps llama.cpp with a REST API. LM Studio wraps it with a GUI. GPT4All wraps it for consumer use. When you type `ollama run deepseek-r1`, you are running llama.cpp. Understanding what llama.cpp can and cannot do helps you debug when these higher-level tools break.
Real benchmarks on my hardware
I compiled llama.cpp from source with CUDA support on my RTX 3090 (24GB). DeepSeek-R1 14B Q4_K_M (8.5GB): 55 tokens/sec with GPU offload, 4 tokens/sec CPU-only. Llama 3.3 70B Q4_K_M (40GB): needs dual GPU or CPU-only at 2 tokens/sec. qwen2.5-coder 7B Q8_0 (7.6GB): 80 tokens/sec GPU, 12 tokens/sec CPU. The Q4_K_M quantization is the sweet spot: 4-bit with medium quality, file size is roughly 0.7 × parameters-in-billions GB. Quality loss from Q8 to Q4 is about 5-8% on benchmarks, barely noticeable in practice.
The GGUF format ecosystem
GGUF is the model file format that llama.cpp uses. It stores the model weights, tokenizer, and metadata in a single file. HuggingFace has 50,000+ GGUF models available for download. The format supports quantization from Q2 (smallest, lowest quality) to Q8 (largest, highest quality). Q4_K_M is the default recommendation for most use cases: good quality, reasonable size. For critical applications where quality matters more than speed, use Q8. For maximum speed on limited hardware, Q3_K_M still produces coherent output.
Server mode for production use
llama.cpp has a built-in HTTP server with an OpenAI-compatible API. Start it with `./llama-server -m model.gguf --port 8080` and any tool that speaks the OpenAI API can use it. I ran this on my HK server to serve a qwen-coder model to multiple developers via Continue.dev. The server handles concurrent requests but does not queue them: if 3 requests arrive simultaneously, all 3 run in parallel on the GPU, which halves throughput. For production, put nginx in front with rate limiting or use a dedicated serving framework like vLLM.
When NOT to use llama.cpp
Do not use llama.cpp if you need: batching (multiple requests sharing one forward pass for higher throughput), PagedAttention or other advanced KV cache techniques, or serving 100+ concurrent users. For those cases use vLLM or TensorRT-LLM. llama.cpp is for running models on consumer hardware, developer machines, and edge devices. It is optimized for low-latency single-user scenarios, not high-throughput production serving.
