Why AI cloud bills surprise teams
Traditional cloud cost management focuses on compute and storage. AI workloads add three new cost dimensions that most FinOps teams have not modelled before: model API calls priced by token, vector database operations priced by query and index size, and data egress driven by retrieval patterns. Miss any one of these and your cost projections will be wrong at scale.
The good news is that AI costs are highly predictable if you instrument them correctly from day one. The problem is that most teams do not instrument them until the first large bill arrives.
The five cost layers in an AI workload
1. Compute
This is the layer FinOps teams know well: EC2, Cloud Run, Lambda, Kubernetes node pools. For AI workloads, the key difference is that compute is often burst-heavy — a document processing job that ingests a batch of files spikes CPU and memory for minutes, then idles. Serverless compute (Lambda, Cloud Run) is usually cheaper for this pattern than always-on instances, but cold starts add latency you need to account for.
GPU compute is a separate category. Fine-tuning or self-hosting a model on GPU instances (p3, g4dn, A100 nodes on GKE) is expensive and should only be considered when API costs at projected volume exceed hosted GPU costs — which typically requires millions of inferences per month.
2. Model API
This is the cost layer that surprises teams most. Model APIs (OpenAI, Anthropic, Cohere, Google) charge per input and output token. A single enterprise workflow can generate thousands of tokens per transaction — system prompts, retrieved context, conversation history, structured outputs — and the cost compounds quickly.
- Audit your prompt templates and measure average token count per call
- Track input vs output token ratio — output is usually priced higher
- Consider using a cheaper model tier for classification or routing steps, reserving the expensive model for generation
- Cache responses for identical or near-identical queries — this is often the single highest-impact cost reduction for FAQ or support use cases
A prompt that passes 8,000 tokens of retrieved context on every call, when only 1,200 tokens are actually relevant, is a cost problem disguised as a retrieval problem.
3. Vector database
Vector databases charge on two axes: index storage (how many vectors you store) and query volume (how many searches per second). For enterprise document sets, index size grows continuously as content is added. For high-traffic applications, query volume can exceed expectations by an order of magnitude if every user interaction triggers multiple retrieval calls.
Cost controls here include: setting namespace or collection limits, archiving stale embeddings on a schedule, batching retrieval where possible, and choosing the right index type (approximate nearest neighbour is faster and cheaper than exact search for most applications).
4. Storage
AI workloads generate a lot of stored data: raw input documents, chunked text, embeddings, model outputs, audit logs and evaluation datasets. S3 and GCS storage is cheap per GB but the volume accumulates. More importantly, data transfer from storage to compute (especially across availability zones or regions) generates egress charges that are not obvious until they appear on a bill.
Place your storage and compute in the same region. Set lifecycle policies to move older data to cheaper storage tiers. Archive raw inputs after embedding if you can reconstruct them from source systems.
5. Data transfer
Egress is the tax that cloud providers charge when data leaves a region or moves between services. For AI workloads that retrieve large documents, stream audio or video for transcription, or sync data between cloud accounts, egress can be 15–25% of total cloud spend. Design your data flow to minimise cross-region movement and use VPC endpoints where available to avoid internet egress charges on internal service communication.
Building an AI cost model before you deploy
Before any AI workload goes to production, build a simple spreadsheet model with these inputs:
- Expected daily active users and average transactions per user
- Average tokens per model API call (input and output separately)
- Average vector queries per transaction
- Average document size and ingestion volume per day
- Data transfer volume between services
Run the model at 1x, 5x and 20x your expected launch traffic. The 20x scenario is not a nightmare case — it is what happens when a pilot gets internal adoption faster than planned. If the 20x cost is acceptable, you are in a healthy position. If it is not, redesign the cost architecture before you deploy.
Tagging and attribution from day one
Every cloud resource in an AI workload should be tagged with at minimum: environment (dev/staging/prod), service name, business unit and cost centre. This allows you to break down spend by workflow, allocate costs to the teams using them and spot waste at the granularity needed to fix it.
Set up a cost anomaly detection alert in AWS Cost Explorer or GCP Cost Management. Configure it to alert on any 20% day-over-day increase in total spend. This catches runaway jobs, accidental resource creation and unexpected traffic growth before they become large bills.
Cost visibility is not a reporting exercise. It is the feedback loop that lets engineering teams make better architecture decisions continuously.