Question 1

How are LLM API costs calculated?

Accepted Answer

LLM APIs charge separately for input tokens (the prompt + context) and output tokens (the generated response). Cost = (Input tokens / 1,000,000) × Input rate + (Output tokens / 1,000,000) × Output rate. Example: Anthropic Claude 3.5 Sonnet at $3/M input and $15/M output. A call with 1,500 input tokens and 400 output tokens costs: (1,500/1,000,000 × $3) + (400/1,000,000 × $15) = $0.0045 + $0.006 = $0.0105/call. At 5,000 calls/day: $52.50/day = $1,575/month.

Question 2

What is a token and how many tokens is a typical message?

Accepted Answer

A token is roughly 4 characters or 0.75 words in English (varies by language and model). Tokenization rules differ across models but follow similar patterns. Rough benchmarks: a 100-word user message ~130 tokens; a 500-word system prompt ~650 tokens; a 1,000-word document chunk ~1,300 tokens; a 500-word generated response ~650 tokens. OpenAI's tiktoken library and Anthropic's Claude tokenizer can count exact tokens for a given string. Context window size (4K to 2M tokens depending on model) determines how much history + document content can fit in one call.

Question 3

How do input vs output token costs differ and why does it matter?

Accepted Answer

Output tokens are consistently 3-5x more expensive than input tokens across all major providers. This is because generating each output token requires a full forward pass through the model, while input tokens are processed in parallel (attention is computed over all input tokens simultaneously). Implication: applications where the model generates long responses (creative writing, detailed analysis) are fundamentally more expensive than applications with short responses (classification, extraction, short answers). Architectural strategies: use cheaper models for generation when quality permits; limit max_tokens aggressively; use streaming to detect when the key answer appears and stop early.

Question 4

Which LLM provider is cheapest for my use case?

Accepted Answer

It depends on the task. As of mid-2025: For simple classification/extraction tasks: Gemini Flash or GPT-4o mini ($0.15/M input) or Haiku ($0.25/M input) are 10-20x cheaper than frontier models. For complex reasoning/coding: Claude 3.5 Sonnet and GPT-4o offer the best quality-per-dollar. For very high volume, cost-sensitive workloads: open-source models (Llama 3, Mistral) self-hosted on GPU instances can reduce per-token cost by 80-95% vs API pricing, with significant infrastructure overhead. The calculator generates a head-to-head cost comparison across all major providers for your specific input/output token ratio and volume.

Question 5

What is prompt caching and how much does it save?

Accepted Answer

Prompt caching (offered by Anthropic and others) stores the KV cache of a repeated prompt prefix, so subsequent calls with the same prefix only pay for new tokens. Anthropic's cache write costs $3.75/M tokens (1.25x standard), cache read costs $0.30/M tokens (10% of standard). For an application with a 10,000-token system prompt and 100 calls/day: without caching, daily input cost = (10,000 + avg_user_tokens) × 100 × rate. With caching: first call pays cache write; remaining 99 calls pay cache read (10% rate) for the cached portion. Savings are substantial for applications with large, stable system prompts — see the Prompt Caching Savings Calculator for detailed analysis.

Question 6

How do I estimate the cost for a RAG (retrieval-augmented generation) application?

Accepted Answer

RAG applications have two token cost components: (1) Retrieval/embedding: vector search returns k document chunks that are injected into context. If each chunk is 500 tokens and you retrieve 5 chunks, that's 2,500 additional input tokens per query. At $3/M = $0.0075 per query just for retrieved context. (2) Generation: the full prompt (system + user message + retrieved chunks) × output rate. For a production RAG app handling 50,000 queries/day with 5,000 average input tokens and 300 average output tokens using Claude 3.5 Sonnet: (5,000/1M × $3 + 300/1M × $15) × 50,000 = ($0.015 + $0.0045) × 50,000 = $982.50/day = ~$29,500/month.

Question 7

What are batch API options and how much do they save?

Accepted Answer

OpenAI offers a Batch API at 50% discount for non-real-time workloads (24-hour turnaround). Anthropic offers Message Batches API at 50% discount. Google Vertex AI has batch prediction at reduced pricing. For offline workloads — document processing, content generation pipelines, evaluation runs — batch APIs cut costs in half with no quality difference. The trade-off: latency (hours instead of seconds). The calculator includes a batch vs real-time comparison for workloads where both are viable.

LLM Cost Calculator

Inputs coming in next batch

Frequently asked questions