5 commands from zero to production API
`pip install fschat; python -m fastchat.serve.controller; python -m fastchat.serve.model_worker --model-path lmsys/vicuna-7b-v1.5; python -m fastchat.serve.openai_api_server --host localhost --port 8000`. Four terminal windows later, you have an OpenAI-compatible API at port 8000. Any tool that works with the OpenAI SDK works with FastChat: just set `openai.api_base = 'http://localhost:8000/v1'`. The setup is more manual than Ollama but gives you more control over batching, GPU memory, and worker count.
Performance on a single GPU vs vLLM
Benchmark on RTX 3090 with Vicuna-7B: FastChat with 1 worker handled 10 concurrent requests at 25 tok/sec each (batch_size=4). vLLM with PagedAttention handled 30 concurrent requests at 23 tok/sec each. FastChat uses continuous batching (not PagedAttention), which is simpler but 30% lower throughput. For single-user or small-team serving (under 20 concurrent users), the difference is negligible. For 50+ concurrent users, vLLM's PagedAttention is necessary.
The Vicuna models: still decent in 2026
Vicuna-7B and 13B were state-of-the-art open chat models in 2023. In 2026, they are outdated but still useful as baselines. Vicuna-7B costs $0.0001 per 1K tokens on cheap cloud GPUs, making it the cheapest way to run a coherent chat model. The quality is about 60% of Llama 3.3 at 10% of the cost. For internal chatbots that handle simple Q&A, Vicuna is cost-effective. For user-facing applications, use Llama 3.3 or DeepSeek instead.
The controller-worker architecture
FastChat's architecture separates concerns: the controller manages workers, workers load and run models, and the API server routes requests. This means you can add and remove GPU workers dynamically without restarting the API. If a worker crashes, the controller routes requests to other workers. This is a production pattern that Ollama (single binary) does not support. For self-hosted AI services that need high availability, FastChat's architecture is superior.
FastChat vs vLLM vs Ollama vs TGI
FastChat: best for single-GPU chat serving, simple setup, Vicuna models, worker architecture. vLLM: best throughput for 50+ concurrent users, PagedAttention. Ollama: simplest setup, one command, best DX for local use. TGI (Text Generation Inference): HuggingFace's serving platform, best ecosystem integration.
