Tag
7 posts tagged #inference
Browse 7 posts tagged Inference. Setup notes, reviews, comparisons, and workflow patterns.
vLLM is a high-throughput, memory-efficient inference and serving engine for LLMs, built in Python under Apache-2.0. Here is how the project is structured.
Replicate is a cloud API for running thousands of open-source machine learning models — image, video, audio, and LLM — billed by compute time, no infra to manage.
Colibri is a pure-C inference engine for frontier mixture-of-experts LLMs with zero external dependencies. It streams expert weights from disk, runs on commodity CPUs, and ships in a tiny binary.
One API to route across top AI models with zero cold starts, per-second billing, and an OpenAI-compatible endpoint. Built on NVIDIA GH200 hardware.
Tokenless is a YC S26 model router that fans out API calls to multiple models, selects the best responder, and cancels the rest — dropping costs by 50 percent with no quality loss.
LLMKube is an open-source Kubernetes operator that manages local LLM inference across NVIDIA, Apple Silicon Metal, and AMD GPUs from a single YAML spec, with an OpenAI-compatible API.
TokenSpeed targets agentic LLM inference with static-compiler parallelism, KV-safe scheduling, and aggressive Blackwell-era performance goals in its preview release.