Ollama

One command and you are running a large language model locally. That is the Ollama pitch. The open-source CLI handles downloads and optimization, with serving wired in. A local REST API plays nice with Cursor, Continue, and Open WebUI. The model library includes Llama 3 and Mistral. Phi and Gemma. CodeLlama and dozens more. Everything runs on your hardware. Zero data leaves the machine. Many local AI setups are built on top of it, and it pairs nicely with LM Studio if you want a GUI alongside the terminal. Runs on macOS or Linux. Windows too, with 8GB of RAM minimum. GPU acceleration works on Apple Silicon and NVIDIA.