/v1/tokenize

re-tokenize

`/v1/tokenize`

The tokenize endpoint /v1/tokenize allows callers to obtain the results of appling a specific model’s tokenizer to a particular input string. Token counts of prompts and completions are included in the responses for non-streaming completion requests, but this endpoint can be helpful for obtaining token counts post-hoc on calls to streaming requests.

HTTP request

POST https://api.featherless.ai/v1/tokenize

Request body

{
  "model": "string",
  "text": "string"
}

Parameters

Parameter	Type	Description
`model`	string	ID of the model to use. Specify the model to use for generating chat completions.
`text`	string	The string to apply the tokenizer to.

Response body

If successful, the response body will contain data with the following structure:

{
  "tokens": "array"
}

Example

Request

curl https://api.featherless.ai/v1/tokenize \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $FEATHERLESS_API_KEY" \
  -d '{
    "model": "GalrionSoftworks/Margnum-12B-v1",
    "text": "Where was the 2020 world series played?"
  }'

Response

{
    "tokens": [
        9241,
        574,
        279,
        220,
        2366,
        15,
        1917,
        4101,
        6476,
        30
    ]
}

Notes

Note that not all strings will have unique representations when tokenized. Therefore disagreement between token counts returned as part of a completion request, and token counts from independent use of this endpoint are not necessarily indicative of a bug.