Retries on a paid call with no cap or backoff

Retries on a paid call with no cap, no backoff, or both.

Rule retry-stormCost audit

Why it costs money

Every retry is a full paid call when the failure happens after the model has done its work, such as a timeout while the response is streaming or a parse error on the output. A loop that retries immediately and forever turns one bad input into an unbounded series of charges.

Retrying a 429 without waiting also makes the rate limit worse: the requests keep arriving, the limit keeps tripping, and every worker doing the same thing adds to the load. The official OpenAI and Anthropic SDKs already retry with backoff by default, so a hand-written loop around them multiplies the attempts.

Example

import OpenAI from "openai";

const openai = new OpenAI();

export async function complete(prompt: string): Promise<string> {
  while (true) {
    try {
      const res = await openai.chat.completions.create({
        model: "gpt-4o",
        max_tokens: 500,
        messages: [{ role: "user", content: prompt }],
      });
      return res.choices[0].message.content ?? "";
    } catch (err) {
      console.warn("OpenAI call failed, retrying", err);
    }
  }
}

import OpenAI from "openai";

const openai = new OpenAI({ maxRetries: 0 }); // retries are handled below
const MAX_ATTEMPTS = 4;

export async function complete(prompt: string): Promise<string> {
  for (let attempt = 1; ; attempt++) {
    try {
      const res = await openai.chat.completions.create({
        model: "gpt-4o",
        max_tokens: 500,
        messages: [{ role: "user", content: prompt }],
      });
      return res.choices[0].message.content ?? "";
    } catch (err) {
      const status = err instanceof OpenAI.APIError ? err.status : undefined;
      const transient = status === 429 || (status !== undefined && status >= 500);
      if (!transient || attempt >= MAX_ATTEMPTS) throw err;
      await new Promise((r) => setTimeout(r, 2 ** attempt * 250 + Math.random() * 250));
    }
  }
}

How to fix it

Retry only errors that can succeed on a second try (429, 5xx, connection resets), wait longer after each attempt with some random jitter, and stop after a fixed number of attempts. Do not retry 400-class errors such as invalid requests or context-length errors: they fail the same way every time.

The simplest option is often the SDK's own setting (maxRetries in the OpenAI and Anthropic clients) with no extra loop around it. If you do write your own loop, turn the SDK retries off so the two do not multiply.

What the audit detects

• LLM calls that the parser places inside a retry loop or a retry decorator. Example code is skipped.
• The same file is checked for backoff (exponential, backoff, jitter, Retry-After, a sleep with a power) and for a cap (max_retries, max_attempts, stop_after_attempt, range(n), an attempt comparison).
• Silent when both are present. A warning when there is no cap, informational when there is a cap but no backoff. Backoff implemented in a shared helper in another file is not visible to it.