Fallback model that never switches back

A fallback to a pricier model that never switches back.

Rule sticky-fallback-modelCost audit

Why it costs money

Falling back to another model when the primary returns a 429 or a provider error is reasonable. The problem is replacing the primary for good. After one transient error, every later call in that process uses the fallback.

In a long-running server that means every user, until the next restart. If the fallback is a larger model or a pricier provider, cost goes up with no error to show for it. If it is a weaker model, quality drops just as quietly.

Example

class Assistant:
    def __init__(self, primary, fallback):
        self.llm = primary
        self._fallback_llm = fallback

    def ask(self, prompt: str) -> str:
        try:
            return self.llm.invoke(prompt)
        except RateLimitError:
            self.llm = self._fallback_llm
            return self.llm.invoke(prompt)

class Assistant:
    def __init__(self, primary, fallback):
        self.llm = primary
        self._fallback_llm = fallback

    def ask(self, prompt: str) -> str:
        try:
            return self.llm.invoke(prompt)
        except RateLimitError:
            # Use the fallback for this call only; the next call tries the primary again.
            return self._fallback_llm.invoke(prompt)

How to fix it

Use the fallback for the failed call only, and leave the primary in place for the next one. If you want to avoid hammering a primary that is down, switch for a bounded window (for example, a few minutes) and then try the primary again.

Log every fallback with the model used, so a spike in fallback traffic shows up before it shows up on the bill.

What the audit detects

• In files with LLM calls that mention a fallback: an assignment that replaces the primary model or client attribute (self.llm, this.model, self._client and similar) with a fallback value.
• Only when the assignment is inside an except or catch block, found by looking back about 12 lines.
• Silent if the primary is assigned back later in the file from an attribute named primary, original, main or default. A restore done through a helper function is not visible to it.