AS.
Back to Blog

Drastically Reducing AWS & LLM Costs for AI Startups

Cost ManagementAWSOpenAIDevOps

Building AI wrappers is easy. Making them profitable is hard. When you are blindly forwarding user input to gpt-4o, your API bills will skyrocket the moment you get traction.

Here is a step-by-step playbook on how I reduce LLM and infrastructure costs for my clients by 60% or more.

1. LLM Semantic Caching

If two users ask variations of the same question ("How do I reset my password?" vs "I forgot my password, how to reset?"), you shouldn't pay OpenAI twice to generate the same answer.

By implementing Semantic Caching (using Redis or a specialized tool like GPTCache), you embed the user's query and do a similarity search against previously answered questions. If the similarity is above 95%, you return the cached response. Cost: $0.00.

2. Smart LLM Routing

Not every task requires the reasoning capabilities of GPT-4.

  • Use GPT-4o / Claude 3.5 Sonnet for complex reasoning, coding, or math.
  • Use GPT-4o-mini / Claude 3 Haiku for classification, formatting, and summarizing.
  • Use Llama-3 (Local via Ollama/AWS EC2) for massive batch processing of sensitive data.

Build a router function in your backend that assesses the complexity of the prompt and sends it to the cheapest model capable of completing it.

3. Optimizing RAG Context Windows

Tokens cost money. If your vector database returns 10 chunks of text, and each chunk is 1000 tokens, you are sending 10,000 tokens to the LLM per query.

Instead of dumping everything, use an LLM-based summary agent to compress the retrieved context before passing it to the final reasoning model. Alternatively, enforce strict chunk sizes and use BM25 keyword filtering to eliminate irrelevant documents before they ever reach the context window.

4. AWS Spot Instances for Batch Jobs

If you are running open-source models on AWS, do not use On-Demand EC2 instances. Use AWS Spot Instances. They are unused EC2 capacities available at up to a 90% discount. While they can be interrupted, if you architect your background workers (using Celery or AWS SQS) to handle retries gracefully, you can process massive AI jobs for pennies on the dollar.