What happened
iroh released Mesh LLM, a distributed AI computing system that pools existing GPUs and memory across multiple machines to run large language models. The software presents an OpenAI-compatible API, enabling local execution, routing to peer nodes, or splitting models across several machines for inference. Mesh LLM supports over 40 models, including 235B mixture-of-experts models, and uses iroh's secure, NAT-traversing QUIC networking to connect nodes. The lightweight 18 MB software allows users to join public or private meshes.
Why it matters
Centralised, metered API providers for LLM inference face a new challenge. Platform engineers and founders gain control over model execution, data locality, and hardware, reducing reliance on external services. Mesh LLM's ability to pool existing, underutilised compute resources directly lowers operational costs by shifting inference from pay-per-use APIs to owned infrastructure. This follows ZML's recent release of its cross-chip AI inference server, intensifying the trend towards distributed, cost-optimised LLM deployment.
