Skip to main content

​
Request limits

A request over a count limit fails with 400 invalid_request and param naming the field. A body over the size limit fails with 413 request_too_large. Duplicate keys anywhere in the JSON body are rejected with 400. These are global ceilings. A model can set lower limits; GET /v1/models reports the effective max_questions, max_options, max_levels and max_body_bytes for each model. Read them there instead of hard-coding them.

​
Input length

Each model also has an input-length limit, in tokens of that model’s tokenizer: The model reads each question separately, together with the state. For every question, the state, that question’s instructions and criteria, and a small fixed prompt the API adds must fit in the limit together. A choice question with many options may be read in smaller groups of options, each with the state. The limit applies to each of these inputs, not to the request as a whole, so adding questions does not bring a request closer to it, but a longer state does. The API never shortens an input to make it fit. A request with an input over the limit fails with 400 invalid_request and no param (the model rejected the input), and is not charged. Shorten the state, or the instructions and criteria of the longest question, and send it again. See Errors. GET /v1/models does not report this limit; this table lists it for each model. A pinned id’s limit never changes. The torchcast-decider-latest alias has the limit of the pinned id it points at.

​
Rate limits

Each API key and each account has a rate limit, in requests per second. The defaults are 10 requests per second per key and 20 requests per second per account, shared by all of the account’s keys. The console shows your current limits under API keys, on the Rate limits tab. If you need a higher account limit, contact sales at contact@torchcast.ai. Limits are counted in one-second windows. Every request counts once the API key is accepted, including requests that are then refused with 400, 410, 413 or 503. A request refused by its key’s own limit does not count against the account. Once the API key is accepted, every response to POST /v1/decisions, success or error, carries the limit that binds (the tighter of the key and account limits) in these headers: A 401, and a body over the global size limit (413), are answered before the key is checked and carry none of these headers. Over the limit, the API answers 429 rate_limited with a Retry-After header: wait that many seconds before the next request. A rate-limited request is not charged and does not reach the model. Throttle on x-ratelimit-remaining to avoid reaching the limit. A separate 429 capacity_exceeded means the model is at capacity, independent of your own limit. It also carries Retry-After. GET /v1/models is public and limited to 5 requests per second per client IP. It does not count against your key or account.

​
Timeouts

The API gives each request about 90 seconds, including the wait for the model. If the model has not answered by then, the request fails with 504 upstream_timeout and is not charged. No request is held open for more than 120 seconds in total. Set your client timeout to 130 seconds, so the API’s own answer, with its x-request-id, always reaches you first. A 504 that arrives without the JSON error body is the same timeout: treat it as upstream_timeout.

​
Retries and idempotency

The API does not support an Idempotency-Key header. Every request is a new request:
  • A failed request is never charged, so retrying after an error costs nothing extra.
  • A request that succeeded is charged again if you send it again. If your client times out before the response arrives, the request may have succeeded and been charged. Keep your client timeout above the API’s own (see above) to avoid that case.
See Errors for which errors to retry.