Reflex
A GGUF-native Rust and CUDA inference engine built for cold starts. Serverless, scale-to-zero LLM and RAG workloads without paying for idle GPU hours. Every CUDA kernel is compiled ahead of time and shipped inside the binary — nothing is JIT-compiled when the process starts.
Overview
Reflex is a local execution engine for single-tenant decisions: one request at a time, no scheduler, no batching, no network server inside the engine. It is built for the moment a serverless GPU worker becomes useful — the time from launching a process to its first result.
Every CUDA kernel is compiled ahead of time by nvcc and shipped inside the binary. There is no CUDA-graph capture and no torch.compile at startup. The result is a cold start measured in hundreds of milliseconds, not minutes.
HTTP lives in a separate OpenAI-compatible sidecar over local IPC. The engine itself offers only sequential local IPC and in-process bindings. That boundary is architectural, not a backlog item: the engine stays a single-shot process, and the platform — Kubernetes, a serverless provider, an autoscaling group — is the orchestrator.
Economics
Most internal AI assistants are quiet: a documentation bot, a support tool handling a few tickets, a daily summarizer. The default deployment is still a GPU server running 24/7 “just in case.” Here is what that readiness costs at 10 questions a day.
| Line item | Always-on RTX A4000 | Reflex serverless |
|---|---|---|
| Rate | $0.25/hr × 24 h | $0.58/hr, pay-per-second |
| Compute per day | $6.00 | ~$0.04 |
| Compute per month | ~$180 | ~$1.20–1.50 |
| GPU energy per month | ~23 kWh (T4 idle at 32 W) | 0.15–0.3 kWh |
| Utilization | >99% idle | Billed only while answering |
Cold-Start Benchmarks
Cold starts measured from outside the process on a Tesla T4 with Qwen3-0.6B-Q4_K_M unless noted. The engine-vs-engine rows time first token. Lower is better.
| Engine | Result | Caveat |
|---|---|---|
llama.cpp llama-simple |
1.13× faster (0.83 s vs. 0.94 s) | Both precompile kernels; this is model-load work |
llama.cpp llama-cli |
1.9× faster (0.83 s vs. 1.58 s) | llama-cli always applies the chat template |
| Ollama | 2.6× faster (0.83 s vs. ~2.1 s) | Cold daemon plus cold model |
| vLLM | 48–150× faster (0.83 s vs. 39.8–127.3 s) | CUDA-graph capture and torch.compile at startup |
| vLLM on Runpod serverless | ~3.4× faster (42–44 s vs. ~150 s) | Mostly platform provisioning; vLLM also downloaded BF16 weights |
| llama.cpp server on Runpod serverless | Engine load ~1.9× faster (0.49 s vs. 0.95 s on L4) | End-to-end inconclusive; platform overhead swamps the gap |
| TypeSafe Jev (managed API) | 1.1–2.0× slower (0.63 s vs. 0.31–0.57 s) | Jev is always warm; Reflex starts a process each time |
f32-weights T4 measurements taken before f16 and quantized-resident weights became the default. On a real Runpod serverless endpoint, Reflex’s own engine load took ~1.6 s of a ~42–44 s cold invocation; the rest is the platform provisioning the GPU and container, which no engine avoids. vLLM’s official Runpod worker took ~150 s on the same GPU tier the same day. Full methodology and per-run numbers: docs/benchmarks.md.
Technical Details
A single Rust binary with CUDA kernels compiled ahead of time. No Python runtime, no PyTorch import, no JIT compilation on the critical path.
Loads GGUF models directly. Weights are held quantized on the GPU by default: 270 MiB resident for Qwen3-0.6B vs. 878 MiB with f16; 4.1 GB for Mistral 7B vs. 13.8 GB.
One request at a time, sequential local IPC. No request queue, no continuous batching, no PagedAttention-style allocation. Run more processes to serve more requests.
Text generation, system1 candidate scoring in one forward pass, text embeddings (Nemotron-3.5-Embed), and decision models (lev, OpenJev) through the sidecar.
A cold reflex system1 on a T4 costs ~5.4 J with idle draw subtracted. Most of the ~28 J the GPU draws over that window is idle power.
HTTP endpoints for /v1/chat/completions, /v1/classify, /v1/embeddings, and llama.cpp’s /v1/systemone live in a separate adapter over local IPC.
Supported Models
Each architecture counted as supported only after its generated tokens matched an independent implementation on real hardware.
| Architecture | Tested with |
|---|---|
| Dense Qwen3 | Qwen3-0.6B |
| Dense Qwen2 / Qwen2.5 | Qwen2.5-0.5B-Instruct; synthetic fixtures |
| Qwen3-MoE | Synthetic fixtures (real tensor layout, top-k routing) |
| Llama / Mistral | TinyLlama-1.1B |
| Qwen3.5 hybrid Gated DeltaNet | Qwen3.5-0.8B; Qwen3.8-27B Q8_0 |
| DeepSeek-V2/V3 MLA | DeepSeek-V2-Lite; GigaChat3.1-10B-A1.8B; GLM-4.7-Flash |
| Granite 4.0-H Mamba-2 hybrid | Granite 4.0-H 350M and 1B Q8_0 |
| Kolibri-1 MoE | Kolibri-1 78B Q4_K_M |
| Gemma 4 dense (text only) | Gemma 4 12B-it and 31B-it Q8_0 |
| Nemotron-3.5-Embed | Nemotron-3.5-Embed-1B BF16/Q8_0/Q4_K_M |
| Decision models: lev, OpenJev | lev Q8_0; llama.cpp’s OpenJev test model |
convert_hf_to_gguf.py. DeepSeek’s MTP heads and group-limited expert routing are not supported. Python bindings (--features python) do not currently build. Full limits: docs/reference.md.
Deployment
For a scale-to-zero endpoint, start with the serverless deployment guide. The job Reflex is built for is a single cold decision, not a warm chat pool.
One-page deployment guide, Runpod first. The recommended starting point.
→ docs/deploy-serverless.mdOne multi-target Dockerfile: reflex, adapter, runpod-lb.
Chat completions, classify, embeddings, and systemone over the sidecar.
→ sidecar/openai-adapterSpot instances and local socket deployment.
→ docs/aws-deployment.mdQueue-based worker, one-click deploy. Optional RAG over your own documents.
→ console.runpod.ioModal Server on an L4 GPU.
→ .modal/README.mdOne Job per invocation; the platform is the orchestrator.
→ docs/reference.mdC FFI (include/reflex_engine.h) and local IPC (reflex stdio / uds).
Honest Caveats
A cold serverless worker takes ~40 seconds, mostly container loading. The engine is fast; the platform is not. For interactive chat, warm servers or always-on setups may still be required.
Serverless platforms keep a pool powered on. Your fractional share of that idle energy is small, but not zero. The comparison above is compute and GPU energy, not total datacenter overhead.
Keeping models and indexes on a network volume avoids re-downloading, but at ultra-low traffic the storage volume fee can exceed the GPU compute cost.
At steady, high volume the picture flips entirely. An always-on server optimized for throughput (like vLLM) becomes the better deal. Reflex deliberately does not batch, schedule, or autoscale.
batch_size is always 1. No network server in the engine. No multi-tenant LoRA router. No internal KV-cache store. No scheduler or autoscaling. These are architectural boundaries, not a backlog.
Getting Started
Browse the source, report issues, and contribute. MIT licensed; contributions need a signed CLA.
→ github.com/lateos-ai/reflexRequires the CUDA toolkit and an NVIDIA GPU. Build with cargo build --release, then run cargo run --release --bin reflex -- smoke.
No GPU of your own? Deploy as a serverless endpoint in one click, no build required.
→ console.runpod.ioSubcommands, build features, kernel modes, JSON output, error categories, Docker, Kubernetes.
→ docs/reference.md