Full chat history re-sent on every turn

The whole conversation re-sent on every turn, so cost grows with its square.

Rule full-history-resendCost audit

Why it costs money

Chat APIs are stateless, so each turn sends the conversation so far as input. If the app appends every message and sends the whole list each time, turn 50 pays again for turns 1 through 49.

As an example, at about 300 tokens per turn, the 50th request carries roughly 15,000 input tokens, and the whole 50-turn conversation adds up to about 380,000 input tokens. Sending a fixed window instead keeps each request roughly the same size, so the total grows in a straight line with the number of turns rather than with its square.

Example

from openai import OpenAI

client = OpenAI()
history = []


def chat(user_text: str) -> str:
    history.append({"role": "user", "content": user_text})
    response = client.chat.completions.create(
        model="gpt-4o",
        max_tokens=800,
        messages=[{"role": "system", "content": SYSTEM_PROMPT}] + history,
    )
    reply = response.choices[0].message.content
    history.append({"role": "assistant", "content": reply})
    return reply

from openai import OpenAI

client = OpenAI()
history = []
MAX_MESSAGES = 20


def chat(user_text: str) -> str:
    history.append({"role": "user", "content": user_text})
    recent = history[-MAX_MESSAGES:]
    response = client.chat.completions.create(
        model="gpt-4o",
        max_tokens=800,
        messages=[{"role": "system", "content": SYSTEM_PROMPT}] + recent,
    )
    reply = response.choices[0].message.content
    history.append({"role": "assistant", "content": reply})
    return reply

How to fix it

Send a bounded window: the last N messages, or as many recent messages as fit in a token budget. Keep the full history in your database if you need it; the model does not have to see all of it.

When older context matters, replace the dropped turns with a short running summary that is updated every few turns. Prompt caching can lower the price of a long, stable prefix, but it does not stop the prefix from growing.

What the audit detects

• Call sites where the parser sees a messages list that is appended to (push, append) and then passed whole to the model call.
• Only when the enclosing function contains no truncation, slicing, windowing or token-budget logic. An app that already trims its history is not flagged.
• Trimming done in a different function or by a framework is not visible to it, so confirm the history really grows without bound.