# 分词 API
> 全部文档的索引见 [llms.txt](https://ark.volcengine.com/llms.txt)；在任一文档页面地址后追加 `.md` 可获取该页的 Markdown 版本。
`POST https://ark.cn-beijing.volces.com/api/v3/tokenization`
对输入的文本进行分词，返回每个 Token 的 ID 和偏移量。仅支持文本输入。
## 鉴权
本接口支持鉴权方式如下，详情请参见 [Base URL 及鉴权](/docs/base-url-and-authentication)。
* API Key 鉴权，请在 [API Key 管理](https://ark.volcengine.com/region:cn-beijing/apiKey) 页面，获取长效 API Key。
---
## 请求参数
### Body 参数
**model** `string` `必选` | 模型 ID
本次请求使用模型的 Model ID 或推理接入点 (Endpoint ID)
**text** `string / string[]` `必选` | 分词内容
需要分词的内容，支持字符串或字符串列表。如需在一次请求中对多条文本分词，请传入字符串列表
**user** `string` | 用户标识
代表最终用户的唯一标识符。当前服务端接受但不使用该字段，预留以对齐其他接口
## 响应参数
**created** `integer` | 创建时间
本次请求创建时间的 Unix 时间戳（秒）
**data** `object[]` | 分词结果
本次请求的分词结果列表，每个元素对应一条输入文本的分词结果
**index** `integer` | 索引
`data.index`
当前元素在 `data` 列表中的索引，与请求 `text` 数组顺序一致
**object** `string` | 对象类型
`data.object`
对象类型，固定为 `tokenization`
**offset_mapping** `integer[]` | 偏移量
`data.offset_mapping`
对文本进行分词后的词语偏移量，列表中每个元素是一个包含两个整数的列表：第一个整数表示词或标记在原始文本中的起始索引（从 0 开始），第二个整数表示结束索引（不包括该索引处的字符）
**token_ids** `integer[]` | Token ID 列表
`data.token_ids`
对文本分词后每个 token 在词表中的 ID 列表，顺序与原文一致
**total_tokens** `integer` | token 总数
`data.total_tokens`
对应内容的总 token 数量
**id** `string` | 请求 ID
本次请求的唯一标识
**model** `string` | 使用的模型
本次请求实际使用的模型 ID（`模型名称-版本`）。例如调用部署 `doubao-seed-2-1-pro-260628` 模型的推理接入点，返回 model 字段信息 `doubao-seed-2-1-pro-260628`
**object** `string` | 对象类型
固定为 `list`
## 代码示例
## 默认
### cURL
#### Request
```bash
curl https://ark.cn-beijing.volces.com/api/v3/tokenization \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $ARK_API_KEY" \
-d $'{
"model": "doubao-pro-32k-241215",
"text": [
"天空为什么这么蓝",
"花儿为什么这么香"
]
}'
```
#### Response
```
{
"object": "list",
"id": "0217441868631536d992e302715172e42d8bf34590a60cadc221f",
"model": "doubao-pro-32k-241215",
"data": [
{
"object": "tokenization",
"index": 0,
"total_tokens": 4,
"token_ids": [14539, 4752, 5189, 5399],
"offset_mapping": [
[0, 2],
[2, 5],
[5, 7],
[7, 8]
]
},
{
"object": "tokenization",
"index": 1,
"total_tokens": 4,
"token_ids": [45018, 4752, 5189, 2920],
"offset_mapping": [
[0, 2],
[2, 5],
[5, 7],
[7, 8]
]
}
],
"created": 1744186863
}
```
### Python
#### Request
```python
import os
from arkruntime import Ark
# Authentication
# 1.If you authorize your endpoint using an API key, you can set your api key to environment variable "ARK_API_KEY"
# or specify api key by Ark(api_key="${YOUR_API_KEY}").
# Note: If you use an API key, this API key will not be refreshed.
# To prevent the API from expiring and failing after some time, choose an API key with no expiration date.
# 2.If you authorize your endpoint with Volcengine Identity and Access Management（IAM),
# set your api key to environment variable "VOLC_ACCESSKEY", "VOLC_SECRETKEY"
# or specify ak&sk by Ark(ak="${YOUR_AK}", sk="${YOUR_SK}").
# To get your ak&sk, please refer to this document(https://www.volcengine.com/docs/6291/65568)
# For more information，please check this document（https://ark.volcengine.com/region:cn-beijing/docs/api-key）
client = Ark(
api_key=os.environ.get("ARK_API_KEY"),
base_url="https://ark.cn-beijing.volces.com/api/v3",
)
print("----- tokenization request -----")
resp = client.tokenization.create(
model="doubao-pro-32k-241215",
text=[
"天空为什么这么蓝",
"花儿为什么这么香",
],
)
print(resp)
```
#### Response
```
{
"object": "list",
"id": "0217441868631536d992e302715172e42d8bf34590a60cadc221f",
"model": "doubao-pro-32k-241215",
"data": [
{
"object": "tokenization",
"index": 0,
"total_tokens": 4,
"token_ids": [14539, 4752, 5189, 5399],
"offset_mapping": [
[0, 2],
[2, 5],
[5, 7],
[7, 8]
]
},
{
"object": "tokenization",
"index": 1,
"total_tokens": 4,
"token_ids": [45018, 4752, 5189, 2920],
"offset_mapping": [
[0, 2],
[2, 5],
[5, 7],
[7, 8]
]
}
],
"created": 1744186863
}
```
### Go
#### Request
```go
package main
import (
"context"
"fmt"
"os"
"github.com/volcengine/ark-runtime-go/arkruntime"
model "github.com/volcengine/ark-runtime-go/arkruntime/model/tokenization"
)
/**
* Authentication
* 1.If you authorize your endpoint using an API key, you can set your api key to environment variable "ARK_API_KEY"
* client := arkruntime.NewClientWithApiKey(
os.Getenv("ARK_API_KEY"),
arkruntime.WithBaseUrl("https://ark.cn-beijing.volces.com/api/v3"),
)
* Note: If you use an API key, this API key will not be refreshed.
* To prevent the API from expiring and failing after some time, choose an API key with no expiration date.
*
* 2.If you authorize your endpoint with Volcengine Identity and Access Management（IAM), set your api key to environment variable "VOLC_ACCESSKEY", "VOLC_SECRETKEY"
* client := arkruntime.NewClientWithAkSk(os.Getenv("VOLC_ACCESSKEY"), os.Getenv("VOLC_SECRETKEY"))
* To get your ak&sk, please refer to this document(https://www.volcengine.com/docs/6291/65568)
* For more information，please check this document（https://ark.volcengine.com/region:cn-beijing/docs/api-key）
*/
func main() {
client := arkruntime.NewClientWithApiKey(os.Getenv("ARK_API_KEY"))
ctx := context.Background()
fmt.Println("----- tokenization request -----")
req := &model.TokenizationRequest{
Text: model.NewStringArrayTokenizationInput([]string{
"天空为什么这么蓝",
"花儿为什么这么香",
}),
Model: "doubao-pro-32k-241215",
}
resp, err := client.CreateTokenization(ctx, req)
if err != nil {
fmt.Printf("tokenization error: %v\n", err)
return
}
fmt.Printf("tokenization response: %v\n", resp)
}
```
#### Response
```
{
"object": "list",
"id": "0217441868631536d992e302715172e42d8bf34590a60cadc221f",
"model": "doubao-pro-32k-241215",
"data": [
{
"object": "tokenization",
"index": 0,
"total_tokens": 4,
"token_ids": [14539, 4752, 5189, 5399],
"offset_mapping": [
[0, 2],
[2, 5],
[5, 7],
[7, 8]
]
},
{
"object": "tokenization",
"index": 1,
"total_tokens": 4,
"token_ids": [45018, 4752, 5189, 2920],
"offset_mapping": [
[0, 2],
[2, 5],
[5, 7],
[7, 8]
]
}
],
"created": 1744186863
}
```
### Java
#### Request
```java
package com.volcengine.sample;
import com.volcengine.ark.runtime.models.tokenization.*;
import com.fasterxml.jackson.core.JsonProcessingException;
import com.volcengine.ark.runtime.service.ArkService;
import java.util.Arrays;
import java.util.concurrent.TimeUnit;
import okhttp3.ConnectionPool;
import okhttp3.Dispatcher;
/*
# pom.xml
com.volcengine
ark-runtime
// 替换正式版本号
0.9.0
*/
public class Sample {
/**
* Authentication 1.If you authorize your endpoint using an API key, you can set your api key to
* environment variable "ARK_API_KEY" String apiKey = System.getenv("ARK_API_KEY"); ArkService
* service = new ArkService(apiKey); Note: If you use an API key, this API key will not be
* refreshed. To prevent the API from expiring and failing after some time, choose an API key
* with no expiration date.
*
* 2.If you authorize your endpoint with Volcengine Identity and Access Management（IAM), set
* your api key to environment variable "VOLC_ACCESSKEY", "VOLC_SECRETKEY" String ak =
* System.getenv("VOLC_ACCESSKEY"); String sk = System.getenv("VOLC_SECRETKEY"); ArkService
* service = new ArkService(ak, sk); To get your ak&sk, please refer to this
* document(https://www.volcengine.com/docs/6291/65568) For more information，please check this
* document（https://ark.volcengine.com/region:cn-beijing/docs/api-key）
*/
static String apiKey = System.getenv("ARK_API_KEY");
static ConnectionPool connectionPool = new ConnectionPool(5, 1, TimeUnit.SECONDS);
static Dispatcher dispatcher = new Dispatcher();
static ArkService service =
ArkService.builder()
.dispatcher(dispatcher)
.connectionPool(connectionPool)
.apiKey(apiKey)
.baseUrl("https://ark.cn-beijing.volces.com/api/v3")
.build();
public static void main(String[] args) throws JsonProcessingException {
TokenizationRequest req =
TokenizationRequest.builder()
.model("doubao-pro-32k-241215")
.text(TokenizationInput.ofList(Arrays.asList("天空为什么这么蓝", "花儿为什么这么香")))
.build();
System.out.println(service.createTokenization(req).toString());
// shutdown service after all requests is finished
service.shutdownExecutor();
}
}
```
#### Response
```
{
"object": "list",
"id": "0217441868631536d992e302715172e42d8bf34590a60cadc221f",
"model": "doubao-pro-32k-241215",
"data": [
{
"object": "tokenization",
"index": 0,
"total_tokens": 4,
"token_ids": [14539, 4752, 5189, 5399],
"offset_mapping": [
[0, 2],
[2, 5],
[5, 7],
[7, 8]
]
},
{
"object": "tokenization",
"index": 1,
"total_tokens": 4,
"token_ids": [45018, 4752, 5189, 2920],
"offset_mapping": [
[0, 2],
[2, 5],
[5, 7],
[7, 8]
]
}
],
"created": 1744186863
}
```