A highly-scalable distributed inference architecture. It features a high-throughput Go gateway orchestrating real-time SSE streams from decoupled PyTorch gRPC workers, paired with Supabase for seamless edge-to-cloud state synchronization.
High-performance backend. Real-time Server-Sent Events (SSE) stream tokens from a Go Gateway, backed by raw gRPC PyTorch workers.
Heavy inference runs locally on isolated Docker nodes, while your chat sessions securely sync across devices via Supabase.
Swap between models like Gemma 3, SmolLM2, and TinyLlama seamlessly. The routing mesh handles the inference pipeline automatically.
The Inference Mesh architecture routes your requests instantly from browser to GPU.
Hot-swap between cutting-edge open-weight models instantly.
Google DeepMind
HuggingFace
TinyLlama Project
* Maximum throughput requires dedicated GPU hardware (CUDA/MPS). CPU fallback will be significantly slower.