conv.

All stories
AIQuiet 7d · day 7

Survey maps self-hosted LLM inference tools for multi-GPU setups

Developer compares LocalAI, exo, GPUStack, vLLM and others for running language models across machines.

newest coverage 6d agosummary written 6d ago3 pieces13 likes & upvotes7 comments MastodonHacker NewsNewswires

What to know

  • The survey categorizes tools by use case: Ollama and Mozilla's offering for single machines, llama.cpp and vLLM for bare engines, and LocalAI, GPUStack, and exo for multi-machine orchestration with distributed inference.
  • LocalAI added distributed mode in June 2026 with peer-to-peer discovery, cache-aware routing, and NATS-based load balancing across nodes.
  • Tools differ significantly in modalities supported (text-only vs. image, video, audio, embeddings), multi-machine capabilities, and whether they can auto-scale to cloud GPUs.

nextime Developer and author

How it unfolded 1 development · click the chart to see its coverage articlesposts

Peak 3 pieces in two hours at Sep 20, 12 PM; 3 pieces over 7 days (1 article · 2 posts) Sep 20, 12 PM — 3 pieces · 1 article · 2 posts — Hacker News 1, Newswires 1, Mastodon 1Sep 20, 2 PM — quietSep 20, 4 PM — quietSep 20, 6 PM — quietSep 20, 8 PM — quietSep 20, 10 PM — quietSep 21, 12 AM — quietSep 21, 2 AM — quietSep 21, 4 AM — quietSep 21, 6 AM — quietSep 21, 8 AM — quietSep 21, 10 AM — quietSep 21, 12 PM — quietSep 21, 2 PM — quietSep 21, 4 PM — quietSep 21, 6 PM — quietSep 21, 8 PM — quietSep 21, 10 PM — quietSep 22, 12 AM — quietSep 22, 2 AM — quietSep 22, 4 AM — quietSep 22, 6 AM — quietSep 22, 8 AM — quietSep 22, 10 AM — quietSep 22, 12 PM — quietSep 22, 2 PM — quietSep 22, 4 PM — quietSep 22, 6 PM — quietSep 22, 8 PM — quietSep 22, 10 PM — quietSep 23, 12 AM — quietSep 23, 2 AM — quietSep 23, 4 AM — quietSep 23, 6 AM — quietSep 23, 8 AM — quietSep 23, 10 AM — quietSep 23, 12 PM — quietSep 23, 2 PM — quietSep 23, 4 PM — quietSep 23, 6 PM — quietSep 23, 8 PM — quietSep 23, 10 PM — quietSep 24, 12 AM — quietSep 24, 2 AM — quietSep 24, 4 AM — quietSep 24, 6 AM — quietSep 24, 8 AM — quietSep 24, 10 AM — quietSep 24, 12 PM — quietSep 24, 2 PM — quietSep 24, 4 PM — quietSep 24, 6 PM — quietSep 24, 8 PM — quietSep 24, 10 PM — quietSep 25, 12 AM — quietSep 25, 2 AM — quietSep 25, 4 AM — quietSep 25, 6 AM — quietSep 25, 8 AM — quietSep 25, 10 AM — quietSep 25, 12 PM — quietSep 25, 2 PM — quietSep 25, 4 PM — quietSep 25, 6 PM — quietSep 25, 8 PM — quietSep 25, 10 PM — quietYesterday, 12 AM — quietYesterday, 2 AM — quietYesterday, 4 AM — quietYesterday, 6 AM — quietYesterday, 8 AM — quietYesterday, 10 AM — quietYesterday, 12 PM — quietYesterday, 2 PM — quietYesterday, 4 PM — quietYesterday, 6 PM — quietYesterday, 8 PM — quietYesterday, 10 PM — quietToday, 12 AM — quietToday, 2 AM — quietToday, 4 AM — quietToday, 6 AM — quietToday, 8 AM — quietToday, 10 AM — quiet 1
Sep 21Sep 22Sep 23Sep 24Sep 25yesterdaynow · 12:54 PM ET
  1. 1

    Developer publishes comprehensive comparison of self-hosted inference orchestrators

    A technical survey compares LocalAI, exo, GPUStack, vLLM, llama.cpp, Ollama, and other tools for running language models on private GPU clusters. The analysis evaluates each tool's multi-machine capabilities, caching strategies, and support for different modalities (text, image, video, audio, embeddings). Star counts are pulled from GitHub as of 2026-09-20.

    “If you have one or more machines with GPUs and want an OpenAI-compatible endpoint in front of them, these are the self-hosted orchestrators that exist in September 2026.”
    — nextime
    1. 1 outlet Self-hosted inference orchestrators compared: LocalAI, exo, GPUStack, vLLM

      first by HN Frontpage, 6d ago