The cheapest GPU hour is the one you don't run. 📉🔋
What does an always-on AI assistant actually cost compared to paying per call? We did the math in dollars and kilowatt-hours—and the results might surprise you.
Most internal AI assistants are quiet: a team's documentation bot, a support tool handling a few tickets, or a script that summarizes data once a day. Yet, the default deployment is a GPU server running 24/7 "just in case."
We wanted to know what that readiness really costs. So, we priced it out on Runpod using Reflex, our open-source inference engine, configured for serverless RAG (retrieval-augmented generation).
The Setup
The Model: A small chat model + embedding model (fits on a single 16 GB GPU).
Option A (Always-On): A dedicated NVIDIA RTX A4000 pod running 24 hours a day.
Option B (Serverless): A worker that spins up on demand, answers the question, waits 5 seconds, and shuts down (pay-per-second).
Traffic: 10 questions a day, spread out. (Prices based on Runpod list prices as of October 24, 2026).
The Money
Always-On: $0.25/hr × 24 hours = $6.00/day (~$180/month).
Serverless (10 questions/day): ~20–30 seconds per call. Works out to about $0.04/day (~$1.20–1.50/month).
That's less than 1% of the always-on bill.
Serverless costs more per hour ($0.58 vs $0.25), but it wins because you don't pay for the hours when nobody is asking questions. Below thousands of daily queries, every idle hour on an always-on server is money spent on waiting.
The Energy
This is where the contrast gets striking:
Always-On: An NVIDIA T4 idles at 32W (before server overhead or datacenter cooling). That's ~23 kWh/month for the GPU alone—roughly a household fridge's daily electricity just to stay ready.
Serverless (10 questions/day): Uses 0.15–0.3 kWh/month (~1% of always-on energy). Ten serverless answers use about as much energy as keeping a phone screen on for 30 minutes.
Why the Engine Matters
Paying per call only works if start-up costs are near zero. If every call wastes a minute loading, the savings evaporate.
That cold-start problem is what Reflex is built to solve. By compiling GPU kernels ahead of time, a Reflex cold start adds just 5.4 joules on top of idle power (~0.0015 Wh). The start-up itself is practically free.
The Honest Caveats
Latency: A cold serverless worker takes ~40 seconds (mostly container loading). For interactive chat, warm servers or always-on setups may still be required.
Shared Idle: Serverless platforms keep a pool powered on. Your fractional share of that idle energy is small, but not zero.
Storage: Keeping models and indexes on a network volume avoids re-downloading, but at ultra-low traffic, the storage volume fee can exceed the GPU compute cost.
High Traffic: At steady, high volume, the picture flips entirely. An always-on server optimized for throughput (like vLLM) becomes the better deal.
The Takeaway
If your assistant answers a handful of questions a day, an always-on GPU spends over 99% of its money and energy waiting. Shifting to a pay-per-second model with an efficient, fast-starting engine cuts both down to roughly 1%.
Reflex's RAG mode is open source and ready to deploy from the Runpod Hub. Turn on one setting and point it at your documents. Full product details and deployment options are on the Reflex product page.
🔗 Check it out on GitHub: https://github.com/lateos-ai/reflex
#AI #MachineLearning #GreenAI #Serverless #LLM #RAG #CloudCosts #Sustainability