Bulk work sent one call at a time

Bulk work sent one call at a time where a batch API costs half.

Rule sync-batch-candidateCost audit

Why it costs money

OpenAI, Anthropic and Google all offer a batch mode that processes requests asynchronously, within a 24-hour window (often much sooner), at half the normal per-token price. Jobs that tag, classify, summarize or embed a backlog rarely need an answer in seconds, so paying the synchronous price for them is a choice rather than a requirement.

Sending items one request at a time also means more requests against your rate limit. Where a module already has a batch method, such as one that embeds a list of texts in a single request, a loop that calls the single-item version bypasses it.

Example

from openai import OpenAI

client = OpenAI()
PROMPT = "Label the review as positive, negative or mixed."


def tag_reviews(reviews: list[dict]) -> None:
    for review in reviews:
        response = client.chat.completions.create(
            model="gpt-4o-mini",
            max_tokens=5,
            messages=[
                {"role": "system", "content": PROMPT},
                {"role": "user", "content": review["text"]},
            ],
        )
        save_label(review["id"], response.choices[0].message.content)

import json
from openai import OpenAI

client = OpenAI()
PROMPT = "Label the review as positive, negative or mixed."


def submit_review_batch(reviews: list[dict], path: str = "reviews.jsonl") -> str:
    with open(path, "w") as f:
        for review in reviews:
            body = {"model": "gpt-4o-mini", "max_tokens": 5, "messages": [
                {"role": "system", "content": PROMPT},
                {"role": "user", "content": review["text"]},
            ]}
            f.write(json.dumps({"custom_id": str(review["id"]), "method": "POST",
                                "url": "/v1/chat/completions", "body": body}) + "\n")
    upload = client.files.create(file=open(path, "rb"), purpose="batch")
    batch = client.batches.create(input_file_id=upload.id,
                                  endpoint="/v1/chat/completions", completion_window="24h")
    return batch.id  # poll client.batches.retrieve(batch.id), then read the output file

How to fix it

If the results are not needed right away, submit the work through the provider's Batch API and collect the results when the batch completes. Keep the synchronous path for anything a user is waiting on.

If the module already has a batch method, route the loop through it. Sending many inputs in one request does not by itself change the per-token price, but it cuts request count and rate-limit pressure, and it is the natural shape for moving to the Batch API later.

What the audit detects

• LLM calls inside a loop that is not in a request handler and not in a retry path, which usually means bulk or offline work. Example code is skipped.
• A stronger finding when the same file already exposes a batch entry point (embed_batch, embed_documents, createBatch, functions ending in _batch or _many, bulk_ functions) that the loop does not use. More than three sibling modules with the same shape are reported as one finding.
• All other loops are summarized in a single low-confidence note for the repo. It cannot tell whether a loop is latency-sensitive or already chunked to the provider's maximum.