Vai al contenuto
AI.info

tools

llama.cpp

Open-source C/C++ software for running large language and vision-language models locally or in the cloud.

llama.cpp

In inglese

llama.cpp runs large language and vision-language models using C/C++ on CPUs, GPUs, mobile devices, and other hardware. It supports command-line tools, quantization, hybrid CPU/GPU inference, and multiple hardware backends.

Developers use it to run models locally, build applications, or launch an OpenAI-compatible API server. The project is software rather than a hosted model service; users provide models and computing hardware separately.

Features

  • Runs LLM and VLM inference in C/C++
  • Provides 1.5-bit through 8-bit integer quantization
  • Supports CPU+GPU hybrid inference
  • Includes command-line tools and a server
  • Offers an OpenAI-compatible API server
  • Supports CUDA, HIP, Metal, Vulkan, SYCL, OpenCL, and WebGPU
  • Provides pre-built binaries and Docker installation
  • MIT License

Use cases

  • Run language models locally on a laptop or workstation
  • Serve models through an OpenAI-compatible API
  • Deploy inference across CPU, GPU, mobile, and embedded hardware
  • Reduce model memory use with quantization
  • Build C/C++ applications with the llama library
  • Test models without sending prompts to a hosted service

Pros

    Cons

      Latest updates

      • b11160

        Adds Vulkan int8 cooperative-matrix quantized matmul for AMD RDNA3 and RDNA4, including q8_0, q4_1, q5_0, and q5_1.

      • b11159

        Handles misalignment in conv_2d and conv_3d.

      • b11158
      • b11157

        Adds CUDA conv3d using implicit GEMM.

      • b11156

        Adds Ling 3.0 VL model support.

      Capabilities

      • Command line — “llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF” source
      • Choice of models — “llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF” source
      • Self-hosted — “locally and in the cloud.” source
      • API — “Launch OpenAI-compatible API server” source

      Get it

      Pricing

      Starting price
      $4/user/mo
      Prices checked
      2026-09-24

      Free

      Free

      • Unlimited public/private repositories
      • Dependabot security and version updates
      • 2,000 CI/CD minutes/month
      • 500MB of Packages storage
      • Issues & Projects
      • Community support

      Team

      • $4 per user/month
      • Access to GitHub Codespaces
      • Repository rules
      • Multiple reviewers in pull requests
      • Draft pull requests
      • Code owners
      • Required reviewers

      Enterprise

      • Starting at $21 per user/month
      • Data residency
      • Enterprise Managed Users
      • User provisioning through SCIM
      • Enterprise Account to centrally manage multiple organizations
      • Environment protection rules
      • Repository rules
      Official website