Reducing LLM Costs in Serverless – A Practical Guide to Caching, Quantization, and Batch Inference

In recent years, language models (LLMs) have become a core component of modern products – from chatbots to recommendation systems. Their potential is growing, but so are the computational costs and low‑latency requirements, especially in serverless architectures where we pay for every millisecond of CPU time. In this article we will show how to combine three proven techniques – caching, quantization, and batch inference – to effectively reduce LLM costs in serverless environments without sacrificing service quality.

Why traditional approaches aren’t enough?

In a serverless model, functions (e.g., AWS Lambda) are invoked on demand and scale automatically. This is convenient, but each invocation means spinning up an environment, loading the model, and processing the request. For large models (tens of GB of memory) the startup and memory costs can exceed the cost of the actual processing. Additionally, the lack of a constantly “warm” process increases latency, which is unacceptable in interactive applications.

Caching LLM results in AWS Lambda

The simplest way to cut costs and latency is caching. If a portion of queries is repetitive (e.g., FAQ questions, documentation look‑ups), you can store the model’s output in a fast store such as Amazon DynamoDB or ElastiCache (Redis). On subsequent function calls you first check the cache – if the answer exists, you return it immediately, bypassing the expensive model invocation.

• The cache key should include both the query text and the model version to avoid mismatches.
• TTL (time‑to‑live) automatically removes stale results, limiting memory usage.
• In a RAG (retrieval‑augmented generation) scenario, the cached value can also include retrieval results, further speeding up the process.

An implementation in Lambda might look like this:

import json, boto3, hashlib

def handler(event, context):
    query = event['query']
    key = hashlib.sha256(query.encode()).hexdigest()
    cache = boto3.client('dynamodb')
    resp = cache.get_item(TableName='LLMCache', Key={'pk': {'S': key}})
    if 'Item' in resp:
        return json.loads(resp['Item']['response']['S'])
    # … model invocation …
    result = run_llm(query)
    cache.put_item(TableName='LLMCache', Item={'pk': {'S': key}, 'response': {'S': json.dumps(result)}, 'ttl': {'N': str(int(time.time())+86400)}})
    return result

Quantization – reducing precision without losing quality

Quantization is a technique that reduces the precision of model weights (e.g., from 32‑bit float to 8‑bit integer). This can shrink the model size by up to 4‑5×, decreasing RAM usage and GPU/CPU requirements. In practice, this means the model can be loaded into a smaller Lambda function (e.g., 2 GB instead of 8 GB) and run faster.

Key trade‑offs:

• Quantization may introduce slight accuracy drops – especially in tasks that require precise numbers (e.g., mathematics).
• Not all frameworks support int8 out‑of‑the‑box; the most common solutions use ONNX Runtime or Hugging Face Transformers with the bitsandbytes extension.
• In a serverless environment it’s worth testing two modes: post‑training quantization (quick, no retraining) and quantization‑aware training (more costly but higher quality).

Example of loading a quantized model in Python:

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model_id = "bigscience/bloom-560m"
model = AutoModelForCausalLM.from_pretrained(model_id, torch_dtype=torch.int8, device_map="auto")
tokenizer = AutoTokenizer.from_pretrained(model_id)

Batch inference in a serverless environment

Batch inference groups several queries and processes them together in a single model call. In a serverless architecture this can be achieved in two ways:

• Aggregation in a queue – e.g., Amazon SQS or EventBridge. A Lambda function collects messages for a short period (e.g., 50 ms) and runs the model on the entire batch.
• Using an Async layer – asynchronous invocations that return identifiers and later retrieve results after the batch completes.

Benefits:

• Better GPU/CPU utilization – the model runs once for a larger number of tokens.
• Reduced number of “cold starts” – a single function handles many requests.
• Lower operational costs – we pay for one invocation instead of many.

Sample code that aggregates queries:

import json, boto3, time

sqs = boto3.client('sqs')
QUEUE_URL = 'https://sqs.region.amazonaws.com/123456789012/llm-requests'
BATCH_SIZE = 8
MAX_WAIT = 0.1

def poll_batch():
    batch = []
    start = time.time()
    while len(batch) < BATCH_SIZE and (time.time() - start) < MAX_WAIT:
        msgs = sqs.receive_message(QueueUrl=QUEUE_URL, MaxNumberOfMessages=1, WaitTimeSeconds=0)
        if 'Messages' in msgs:
            batch.append(msgs['Messages'][0])
    return batch

def handler(event, context):
    batch = poll_batch()
    queries = [json.loads(m['Body'])['query'] for m in batch]
    results = run_llm_batch(queries)  # single model invocation
    for msg, res in zip(batch, results):
        sqs.delete_message(QueueUrl=QUEUE_URL, ReceiptHandle=msg['ReceiptHandle'])
        # optionally send response to another queue
    return {'processed': len(batch)}

Combining Three Techniques – End‑to‑End Architecture

The biggest savings are achieved by combining cache, quantization, and batch inference. A typical serverless flow looks like this:

1. A request hits API Gateway and is placed in an SQS queue.
2. A Lambda function starts, pulls a batch of requests, and first checks the cache (Redis/DynamoDB).
3. Requests not found in the cache are grouped and sent to a quantized model (e.g., ONNX Runtime in int8 mode).
4. Results are written to the cache and returned to the client (e.g., via SNS or directly over HTTP).

This structure reduces the number of full model invocations, lowers the memory footprint required to run the function, and enables better resource utilization through batch inference.

Practical Checklist for Product Teams

• Query analysis: Identify repetitive patterns and determine which ones are worth caching.
• Model and format selection: Use a model available in ONNX format or a ready‑made int8 version.
• Cache implementation: Set up DynamoDB or Redis with appropriate TTL and a hash key based on the query content.
• Batching strategy: Define the maximum batch size and maximum wait time – tune to latency SLA.
• Monitoring and alerts: Track cache hit‑rate, average inference time, and Lambda costs.
• A/B testing: Compare the optimized version against a baseline in terms of cost and response quality.

Common Mistakes and Trade‑offs

When implementing, it’s easy to fall into several traps:

• Overly aggressive TTL: Too short a cache lifetime leads to frequent refreshes and loss of benefits.
• Batch too large: Increases latency of individual requests – balancing cost and SLA is crucial.
• Inappropriate quantization: Quantizing a model to int4 can dramatically speed up inference, but the risk of hallucinations rises.
• Lack of cost monitoring: Without regular AWS bill analysis, it’s easy to miss that extra SQS queue costs can outweigh cache savings.

“Optimizing LLMs in serverless is not a one‑off task, but a continuous process of balancing cost, latency, and quality.”

Conclusion and Invitation to Collaborate

Reducing LLM cost and latency in a serverless architecture requires a conscious combination of cache, quantization, and batch inference. This enables product teams to scale intelligent functions while maintaining budget control and user experience. If you need assistance designing such an architecture, optimizing models, or implementing automation at Coderia.it, get in touch – we’ll help turn your ideas into efficient, cost‑optimized solutions.