Frontier model used for a short, simple task

A frontier model doing a short, simple job where a cheaper sibling is likely enough.

Rule model-oversizedCost audit

Why it costs money

Classification, routing, extraction and yes/no checks produce a handful of output tokens, but you still pay the frontier input rate on the whole prompt. Smaller siblings in the same family often charge a small fraction of that per token.

These calls also tend to be the high-volume ones: every message gets classified, every document gets tagged. At 100,000 calls a day, a price difference that looks negligible per call becomes a large share of the bill.

Example

import OpenAI from "openai";

const openai = new OpenAI();

export async function classifyIntent(message: string) {
  const res = await openai.chat.completions.create({
    model: "gpt-4o",
    max_tokens: 5,
    messages: [
      { role: "system", content: "Reply with one word: billing, bug, feature or other." },
      { role: "user", content: message },
    ],
  });
  return res.choices[0].message.content?.trim().toLowerCase();
}

import OpenAI from "openai";

const openai = new OpenAI();

export async function classifyIntent(message: string) {
  const res = await openai.chat.completions.create({
    model: "gpt-4o-mini",
    max_tokens: 5,
    messages: [
      { role: "system", content: "Reply with one word: billing, bug, feature or other." },
      { role: "user", content: message },
    ],
  });
  return res.choices[0].message.content?.trim().toLowerCase();
}

How to fix it

Run a few hundred real inputs through the cheaper sibling and compare its labels with the current model's. If agreement is high enough for the job, switch. Short output does not always mean an easy task, so measure before changing.

If only some inputs need the larger model, route: send everything to the small model first and escalate when it is unsure, or use a router such as RouteLLM.

What the audit detects

• Calls where the model name is a recognizable frontier tier (gpt-4o, gpt-4-turbo, gpt-4, o1 or o3, Claude Opus or Sonnet, Gemini Pro or Ultra) and a cheaper sibling can be named.
• Only calls with max_tokens set between 2 and 256. A short, capped output is the signal for classification or extraction.
• Calls with no output cap, or a cap above 256, are not flagged, because those usually mean open-ended generation. Calls whose model name cannot be read from the source are skipped.

Research

FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance. Cascading from cheaper to more expensive models matched the best single model's accuracy at up to 98% lower cost on the paper's evaluation datasets.