tools
llama.cpp
Open-source C/C++ software for running large language and vision-language models locally or in the cloud.
llama.cpp runs large language and vision-language models using C/C++ on CPUs, GPUs, mobile devices, and other hardware. It supports command-line tools, quantization, hybrid CPU/GPU inference, and multiple hardware backends.
Developers use it to run models locally, build applications, or launch an OpenAI-compatible API server. The project is software rather than a hosted model service; users provide models and computing hardware separately.
Features
- Runs LLM and VLM inference in C/C++
- Provides 1.5-bit through 8-bit integer quantization
- Supports CPU+GPU hybrid inference
- Includes command-line tools and a server
- Offers an OpenAI-compatible API server
- Supports CUDA, HIP, Metal, Vulkan, SYCL, OpenCL, and WebGPU
- Provides pre-built binaries and Docker installation
- MIT License
Use cases
- Run language models locally on a laptop or workstation
- Serve models through an OpenAI-compatible API
- Deploy inference across CPU, GPU, mobile, and embedded hardware
- Reduce model memory use with quantization
- Build C/C++ applications with the llama library
- Test models without sending prompts to a hosted service