Process requests in batches

A batch is a file with many complete requests that InferNode runs on its own. You upload the file, create the batch, check its status from time to time and download the results. If any line failed, there is also a separate file with those lines.

It makes sense when you have work you do not need right now: classifying historical records, summarizing thousands of documents, generating the vectors of a corpus. Batch lines do not use up your requests per minute, so the heavy work does not leave your agents and workflows without service.

Before you start

• An InferNode key (Use the API from code).
• A model from the Catalog that supports batches. Whoever administers the platform decides which ones, and with how much capacity; if the model does not support them, the file fails with model_not_batchable.
• Batches enabled in your installation. If they are not, /v1/batches responds 404. Check it with any request to that path.

The input file

A text file with one request per line, in JSONL format. Each line carries an identifier of your own (custom_id), the method, the path and the body you would send to the regular endpoint.

{"custom_id": "doc-1", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "Saptiva Turbo", "messages": [{"role": "user", "content": "Resume este expediente en una frase."}], "max_tokens": 64}}
{"custom_id": "doc-2", "method": "POST", "url": "/v1/chat/completions", "body": {"model": "Saptiva Turbo", "messages": [{"role": "user", "content": "¿Qué área firma este documento?"}], "max_tokens": 64}}

For vectors, the lines point to /v1/embeddings:

{"custom_id": "parrafo-1", "method": "POST", "url": "/v1/embeddings", "body": {"model": "<modelo-de-vectores>", "input": ["primer párrafo", "segundo párrafo"]}}

Name the model by its alias (Saptiva Turbo), not by its exact Catalog name: a batch can take a while, and if the model is updated meanwhile, the exact name stops resolving and the remaining lines end in model_not_found. GET /v1/models/aliases lists the ones in your installation.

File rules:

• A single endpoint and a single model. A line that names another path fails the file with url_mismatch, and one that names another model, with model_mismatch.
• Unique custom_id, not empty and at most 512 characters long. It is what lets you match each result with its input, because the output order is not guaranteed.
• No stream. A line with "stream": true fails with stream_not_supported. A batch never responds in streaming.
• The body is the same one the regular endpoint would accept, and it is validated the same way.

The four steps

BASE=https://api.<tu-dominio>/infernode/v1

# 1. Upload the file. Returns a file-... id
curl -sS $BASE/files \
  -H "Authorization: Bearer sk-saptiva-..." \
  -F purpose=batch -F file=@entrada.jsonl

# 2. Create the batch. Returns a batch_... id in validating status
curl -sS $BASE/batches \
  -H "Authorization: Bearer sk-saptiva-..." -H "Content-Type: application/json" \
  -d '{"input_file_id":"file-...","endpoint":"/v1/chat/completions","completion_window":"24h"}'

# 3. Check until the status is completed, failed, cancelled or expired
curl -sS $BASE/batches/batch_... -H "Authorization: Bearer sk-saptiva-..."

# 4. Download the results, with the output_file_id returned by the check
curl -sS $BASE/files/file-.../content \
  -H "Authorization: Bearer sk-saptiva-..." -o resultados.jsonl

completion_window only accepts "24h" and is stored for compatibility with the OpenAI SDK: it is not a delivery promise. A batch advances at the pace allowed by the model's free capacity, which is shared with the rest of the Client.

If you upload a file with the same content again, InferNode returns the one it already had instead of duplicating it, with the X-Saptiva-Deduplicated-From header. The same happens when you create the same batch twice while the first one is still alive.

With the OpenAI SDK

The process is identical to OpenAI's; only base_url changes.

import os, time
from openai import OpenAI

cliente = OpenAI(
    base_url="https://api.<tu-dominio>/infernode/v1",
    api_key=os.environ["SAPTIVA_API_KEY"],
)

archivo = cliente.files.create(file=open("entrada.jsonl", "rb"), purpose="batch")
lote = cliente.batches.create(
    input_file_id=archivo.id,
    endpoint="/v1/chat/completions",
    completion_window="24h",
)

terminales = {"completed", "failed", "expired", "cancelled"}
while (lote := cliente.batches.retrieve(lote.id)).status not in terminales:
    time.sleep(30)

print(lote.status, lote.request_counts)
if lote.output_file_id:
    print(cliente.files.content(lote.output_file_id).text)
if lote.error_file_id:
    print(cliente.files.content(lote.error_file_id).text)

Check every 30 seconds or more: each check is a regular request and it does use up your per-minute limits.

Statuses

Status	What it means
validating	Just created. The file is being reviewed. No line has run
in_progress	The lines are running
finalizing	They finished; the output files are being assembled. It can no longer be canceled
completed	Finished. The results file is available and, if there were failures, the errors file
failed	A problem with the whole file, for example a malformed line or a model that does not support batches. No line ran, and errors lists up to 100 causes
cancelling	You asked to cancel it. Lines in progress finish and are charged
cancelled	Canceled. The results of what did run are kept
expired	The expires_at date was reached. What managed to run is kept

While it advances, request_counts keeps the count of total, completed and failed. failed also includes canceled and expired lines, so a batch you canceled shows there the lines that did not get to run.

The results

The output file has one line for each line that finished successfully, with your custom_id and the response that the regular endpoint would have returned:

{"id": "batch_req_...", "custom_id": "doc-1", "response": {"status_code": 200, "request_id": "req_...", "body": {"choices": [{"message": {"role": "assistant", "content": "..."}}]}}, "error": null}

Everything else goes to the errors file, which only exists if some line failed: error_file_id comes empty when all of them worked. It holds two forms. A line rejected by the model or by validation keeps the code and body that the regular call would have returned, so you can read its cause the same way as in a single call. A line stopped by the batch itself, for example when you cancel it, carries an empty response and an error with its cause.

Between the two files there is always the total number of input lines, so no line is lost without explanation.

Files are not kept forever. Both the input file and the output files expire after 30 days by default, and each file carries its date in expires_at. Download the results you want to keep.

Files are downloaded only with GET /v1/files/{id}/content, which responds with the X-Saptiva-Content-SHA256 header so you can verify what you downloaded. There are no temporary download links.

Cancel

curl -sS -X POST $BASE/batches/batch_.../cancel -H "Authorization: Bearer sk-saptiva-..."

The batch moves to cancelling. What was already running finishes and is charged; what had not started appears in the errors file as batch_cancelled and is not charged. That way every line is accounted for. A batch in finalizing or already finished responds 400 with batch_not_cancellable.

An administrator of your organization, or whoever operates the platform, can also cancel a batch of yours. When that happens, the batch carries an entry in errors.data: cancelled_by_org_admin (it names the person) or cancelled_by_platform_admin. If you cancel it yourself, no entry appears.

What it costs and what it consumes

• Each line costs the same as the equivalent call to the same model. There is no batch discount.
• Usage is recorded separately from interactive usage: in GET /v1/billing/balance, the last movement (last_event.endpoint) carries the marked endpoint, for example /v1/chat/completions:batch.
• Lines do not use up your requests per minute. The control calls do: uploading, creating, checking, listing and canceling.
• If the balance runs out halfway through the batch, the remaining lines fail with insufficient_credits and are recorded in the errors file.

File limits

Limit	Default value	If exceeded
File size	100 MiB	413 file_too_large when you upload it
Line size	1 MiB	The file fails with invalid_json_line
Lines per file	50,000	The file fails with too_many_tasks
Vector inputs, adding up all lines	50,000	The file fails with too_many_tasks
Client storage	10 GiB	400 storage_quota_exceeded when you upload
Active batches at once	Depends on the plan	429 too_many_active_batches
Queued lines	Depends on the plan	The file fails with enqueued_request_limit_exceeded

Output files count toward storage but are never rejected. If you run out of space, delete files you no longer use with DELETE /v1/files/{id}. An input file used by a live batch cannot be deleted, and the request responds file_in_use.

When something fails

What you see	Cause	What to do
The batch stays in failed without anything running	A problem with the whole file. errors.data lists up to 100 causes with their line	Fix the file and upload it again
model_not_batchable	That model has no capacity assigned for batches	Ask for capacity to be assigned, or use another model
model_not_found	The model does not exist for your key, or it was retired halfway through the batch	Check GET /v1/models
duplicate_custom_id	Two lines share an identifier	Make the custom_id values unique
413 file_too_large	The file is larger than 100 MiB	Split it into several batches
429 too_many_active_batches	You already have as many batches running as your plan allows	Wait for them to finish
authorization_revoked in the lines	The key, the membership or the organization stopped being valid while it ran	Check the key and create the batch again
It stays in validating for a good while	The startup retries on its own. It can take up to about half an hour to start	Wait; do not create it again, because that would return the same batch
in_progress without advancing	The model's capacity is busy with other work from the Client	This is normal in large batches. If it does not advance in hours, report it with the batch identifier

Next step

• Use the API from code
• InferNode API
• Limits and quotas