Implementing Prompt Compression to Reduce Agentic Loop Costs

Implementing Prompt Compression to Reduce Agentic Loop Costs

Agentic loops in production face quadratic cost accumulation as context grows with each step. Prompt compression techniques like recursive summarization can reduce a 109-token context to 36 tokens, yielding 67% savings.

Why This Matters

In production, agentic frameworks like LangGraph require maintaining context across multiple steps, causing token counts to explode quadratically rather than linearly. This creates a financial burden and increases inference latency, as long prompts take significantly longer to process and increase compute overhead.

Key Insights

• Agentic loop costs grow quadratically as context is prepended to each subsequent step (Iván Palomares Carrascosa, 2026).
• Instruction distillation achieves shorthand prompts that models interpret identically to full prose, saving significant token overhead.
• Recursive summarization periodically condenses step history using smaller LLMs like Llama 3 or GPT-4o-mini.
• Local vector databases such as FAISS or Chroma can replace full history by retrieving only relevant actions via RAG.
• The LLMLingua framework optimizes prompts by stripping stop words and repetitive JSON structures before sending to larger models.

Working Examples

A Python example demonstrating recursive summarization and instruction distillation to achieve 67% token savings.

import tiktoken\ndef count_tokens(text, model="gpt-4o"):\n    encoding = tiktoken.encoding_for_model(model)\n    return len(encoding.encode(text))\ndef compress_history(history_list):\n    print("--- Compressing History ---")\n    combined = " ".join(history_list)\n    summary = f"Summary of {len(history_list)} steps: Tasks A & B completed. Result: Success."\n    return summary\nsystem_prompt = "Act: ResearchBot. Task: Find X. Output: JSON only. Constraints: No fluff."\nhistory = []\nfor step in range(1, 6):\n    action = f"Step {step}: Agent performed a very long-winded search for data point {step}..."\n    history.append(action)\n    current_full_context = system_prompt + " ".join(history)\n    raw_tokens = count_tokens(current_full_context)\n    print(f"Loop {step} | Full Context Tokens: {raw_tokens}")\ncompressed_context = system_prompt + compress_history(history)\ncompressed_tokens = count_tokens(compressed_context)\nprint(f"Final Compressed Tokens: {compressed_tokens}")

Practical Applications

• Use Case: ResearchBot implementing distilled system prompts to save 3,000 tokens over a 100-step loop. Pitfall: Linear prompt growth causing cost explosions in long-lasting agentic loops without compression.
• Use Case: Production agents using recursive summarization to reduce 500K token contexts to 32K windows. Pitfall: High latency and compute overhead caused by sending redundant context tokens repeatedly.

References:

• https://machinelearningmastery.com/implementing-prompt-compression-to-reduce-agentic-loop-costs/