语言及多模态理解模型精调数据格式

1
2
3
4
5{"messages":[{"role":"system","content":"请描述图片中的物体及场景"},{"role":"user","content":[{"type":"image_url","image_url":{"url":"tos://bucketname/image/desk.png"}},{"type":"text","text":"这张图片里有什么？"}]},{"role":"assistant","reasoning_content":"用户让我描述图片中的物体和场景，首先看图片内容：画面中有一张木质书桌，上面放着笔记本电脑、咖啡杯和绿植，背景是白色墙壁和窗户，光线明亮。需要准确列出物体，描述场景氛围，保持简洁明了。","content":"图片中展示了一个整洁的办公场景：一张木质书桌上放置着笔记本电脑、白色咖啡杯和一盆绿色多肉植物，背景是带有窗户的白色墙壁，自然光线充足，整体氛围清新舒适。","loss_weight":1.0}],"extra":{"sample_id":"sample-001","metadata":{"source":"user"}},"thinking":{"type": "enabled"}}
{"messages":[{"role":"user","content":[{"type":"text","text":"计算图片中物体的数量"},{"type":"image_url","image_url":{"url":"data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAMCAgICAgMCAgIDAwMDBAYEBAQEBAgGBgUGCQgKCgkICQkKDA8MCgsOCwkJDRENDg8QEBEQCgwSExIQEw8QEBD/2wBDAQMDAwQDBAgEBAgQCwkLEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBD/wAARCABAAEADAREAAhEBAxEB/8QAHQAAAgEFAQEAAAAAAAAAAAAAAAgGAQMEBwkFAv/EADgQAAEDAwIDBAcFCQAAAAAAAAECAwQABQYHERIhMQgTQVEJFDJhcYGRFRYiQlMzNlJydIKSobP/xAAcAQABBAMBAAAAAAAAAAAAAAAABAUGBwEDCAL/xAA0EQABAwMCAwUFCAMAAAAAAAABAAIDBAURBiESMUEHUWFxoRMyUpGxFBUiIzSBwdFCcvD/2gAMAwEAAhEDEQA/AOqdCEUIVCQBuTsBQhYZvdlDvcG7wu86cHrCOL6b1u+zTY4uA48ivXA7uWYCCAQQQehFaV5VaEIoQihCKELz79fbZjNnl368SUsQ4TRddWfADwHmT0A8zSilpZa2ZtPCMuccBemMMjg1vNJXqjr3l+oM56PDmv2uyhRSzEYWUlafNxQ5qJ8ugq9bBo+itEYfM0Pl6k8h5BSKnoo4Rk7lax4lcXHxHi89+dS7gbjhxsleAp/gnaDzHStC5TsiRd7NHQXH7e84VHgHM90o+yrboOh8ajF40ZRX3aMCOU8nDlnxHcm64wRiF0oG4GU6On+f4tqdicDM8Nubc62XBvjQtJ/EhX5kLT1StJ5EGqMutqq7LVvoq1nC9p3H8jvB6FRyGZlQwSRnIKkVNy2ooQihCXPthZRIi2my4lHcUlE5xcuQAfaS3sEA/wBxJ+QqzOze3tlqJaxw9wYHmefona1xguLz0SsVcKel9NtuOrDbTalrV0Skbk/IV5e9sY4nHAWCQOa9ZzT/ADSfAdLWH3l1l1tSeJMFwggj4Ug+/LdFIA6doIPxBaJpI3RuaSNwtPdnjtB5t2Zs1cTJhzHsdmP93drS8lSOIA7d62Fey4kfUcj7pfrHSNt13Qh8D2+3aPwuBB/Y46H0VNUF0FHMWscC3O4yur2DZxjGo+Lwcxw+6tXC13BsONOoPMHxQodUqB5EHmDXIdztlVZ6p9FWMLZGnBB+o7wehU4hmZOwSRnIK96kC2ooQlR7ZH70Y9/QO/8ASrg7M/00/wDsPonu1e47zS/WCJKyrN7Lp7YkB+8Xp7hQj8rDI5uPueSUpBPvPIVY1wqG223y3GbZjB8yeTR4krNfdI6NzYhu93IfyV0BwbSzC9P4LUax2dj1hKAHZjiAp51W3MlR5jfyHKucbpfa67yF9Q846DoP2TVLUSTHLipaehpnCTu5JIsrtFqulyuMS5W+NMYMl0cD7SVgjjPmKlNHX1VEQ+nkLT4EhcdXKolprjMYXEYe7kfEq5o68NFcjcm4e47HslxWDcrMVlUZw/qtA/s3B5jkRyI6EOd4vk2oIAy5Djkb7r/8vI9489wplpntErbVKI6v8yM8+8eKdaO+3JYbktHdDqAtJ9xG4qCEYOF0jFI2VgkbyIz81crC9pPO3ff4uMTbPeJfMM253gR4rWXNkpHxNXV2TUj61s0LOrh8sJbHXMt9JJO/p6lQb0a9idybMM51SvI76XHaYtrDh6I70lawny5IQPhT7221QoaajtEOzTl58cbDPqolaZZK2eSrmOSU/lc8KQK3JdSxHdfWQEtoUok+AA3rI3K1zPEcbnnoCkomO+sTH3/1HVr+pJp+bsFxdWSe2qJJO8k+qy8fs8m/3uFZ4iCpyW8lsbDoN+Z+AG5ry9wY0kpVZ7fJdK+KkiG7iB/ZTmR2URo7UdsbJaQlCfgBtTGTk5XY8UYijbGOgAVysLYuePpQrhJGXYTawohg22Q+R/ErvQB9Bv8AWuk+weFhpauU8+Jo9FGtQzvDWQDlzUk9F3kME2bOcVK0iamVFuCU781NFCkE/Igf5Cmrt3pJBV0lXj8Ja5v7g5+izp144Hs6809VUCpIolqpfkY/g1zlcYS6+0YzI35la+X+huflW6nZxyAKK60ubbVZJ5SdyOEeZ2SnMsuyHUR47SnHHCEoQkblRPQAU8kgblcoRRSVDxHGMuPIBMdo7pcvFGPvBfGx9qyEbIb6+roPh/MfHy6U2VNR7Q8LeS6M0Bov7ij+3Vg/OcNh8I/s9VtCkas1FCEqnb60BverGD2/McPgrmXvFe9UuI2N3JMRexWEDxUkpCgPEcXjVtdkuroNO3F9JWu4YpsDPQOHInwPJMt6onVMQezm36LnfpVqnmeimbx8zxCR6vcInEy+w8k92+2T+Nl1PI7Hb4ggEcxXTOoNP0Gq7eaOr3YdwRzB6EH/ALKilNUyUcnGzmnMielHgfZqBO0hkquHAAsM3RIZK/dujiA38OdUZL2ETiU8FY3g7y05x88KQDUTQ3Lmb+a2Jit21v7RVpiXfJsai2CA8538WKkLShhsjYKcWrm4sjnsAAN+nWqwv9Fa7JVmkoJTJw7FxxueuAOgVbaggvOvKltNSs4adh947Anv8ceC3ngWk+P4SlMsj165bfikuJ9jzCE/l+PWotNUOl26Kc6Y0Pb9OASAccvxHp5DopxSdTVFCEUIRQhao1C7LOg+p9wXd8s0+gruDp3clxFLiuuHzWWiniPvO5qW2fXN/sUfsqKpcGDocOA8gc4SKe3U1QeJ7d1h4T2Q+zzgM9F1senEJ2Y0Qpt6e4uWUEHcFIdKgD79t63XTtB1HeIzFU1TuE8w3Dc+fDheYrXSwnLWfPdbiSlKEhCEhKQNgANgKhhOdylzWhow0bKtCyihCKEL/9k="}},{"type":"image_url","image_url":{"url":"file:./image/books.png"}}]},{"role":"assistant","content":"图片中包含3个主要物体：1台笔记本电脑、1个咖啡杯、1盆绿植，共3个物体。"}]}
{"messages":[{"role": "user","content": [{"type": "image_url","image_url": {"url": "file:./image/doubao_logo.jpeg"}},{"type": "text","text": "What’s in this image?"}]},{"role": "assistant","content": "It is the logo of Doubao LLM model."}]}
{"messages":[{"role":"user","content":[{"type":"text","text":"请将这段客服通话转写为文字，并总结客户诉求。"},{"type":"audio_url","audio_url":{"url":"tos://bucketname/audio/customer_call.wav"}}]},{"role":"assistant","content":"转写：客户咨询订单配送延迟的问题，并提供了订单号。总结：客户希望确认延迟原因和预计送达时间。","loss_weight":1.0}]}
{"messages":[{"role":"user","content":[{"type":"text","text":"请根据视频画面和声音判断这段素材是否适合用于产品介绍。"},{"type":"video_url","fps":1,"video_url":{"url":"tos://bucketname/video/product_intro.mp4"}}]},{"role":"assistant","content":"这段视频画面清晰，讲解声音完整，适合用于产品介绍。","loss_weight":1.0}]}

1
2{"messages":[{"role":"system","content":[{"type":"text","text":"请根据古诗内容，仅回复作者的名字。"}]},{"role":"user","content":[{"type":"text","text":"姑苏城外寒山寺，夜半钟声到客船"}]},{"role":"assistant","reasoning_content":"用户让我根据古诗内容回复作者名字，诗句是“姑苏城外寒山寺，夜半钟声到客船”。这句诗出自张继的《枫桥夜泊》。用户要求仅回复作者名字，所以直接回复。","content":"张继","loss_weight":1}]}
{"messages":[{"role":"user","content":[{"type":"text","text":"什么时候需要 SFT？"}]},{"role":"assistant","content":"当 prompt engineering 无法稳定解决问题，或模型输出需要满足固定格式但仍有部分 case 不符合要求时，可以考虑使用 SFT。SFT 也可用于减少 prompt 中的复杂约束，降低线上推理耗时。","loss_weight":1}]}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49{
    "messages": [
        {
            "role": "system",
            "content": "请描述图片中的物体及场景"  // 本示例中system角色使用文本内容
        },
        {
            "role": "user",
            "content": [  // user角色支持数组格式，混排文本、图片、音频、视频
                {
                    "type": "image_url",  // 内容类型：图片
                    "image_url": {
                      "url": "tos://bucketname/image/desk.png"  // 图片地址（TOS/base64/相对路径）
                    }
                },
                {
                    "type": "text",  // 内容类型：文本
                    "text": "这张图片里有什么？"
                },
                {
                    "type": "audio_url",  // 内容类型：音频
                    "audio_url": {
                      "url": "tos://bucketname/audio/customer_call.wav"  // 音频地址（TOS）
                    }
                },
                {
                    "type": "video_url",  // 内容类型：视频，支持视频内嵌音频
                    "fps": 1,  // 视频抽帧帧率，表示每秒抽取的帧数
                    "video_url": {
                      "url": "tos://bucketname/video/product_intro.mp4"  // 视频地址（TOS 或 Base64）
                    }
                }
            ]
        },
        {
            "role": "assistant",
            "reasoning_content": "用户让我描述图片中的物体和场景，首先看图片内容：画面中有一张木质书桌，上面放着笔记本电脑、咖啡杯和绿植，背景是白色墙壁和窗户，光线明亮。需要准确列出物体，描述场景氛围，保持简洁明了。",
            "content": "图片中展示了一个整洁的办公场景：一张木质书桌上放置着笔记本电脑、白色咖啡杯和一盆绿色多肉植物，背景是带有窗户的白色墙壁，自然光线充足，整体氛围清新舒适。",
            "loss_weight": 1.0  // 训练权重，默认1.0
        }
    ],
    "extra": {
        "sample_id": "sample-001",
        "metadata": {
            "source": "user"
        }
    },
    "thinking": {"type": "enabled"}  // 深度思考能力控制字段
}

1{"messages":[{"role":"system","content":[{"type":"text","text":"你是一名多模态质检助手，需要结合文本、图片、音频和视频判断回复质量。"}]},{"role":"user","content":[{"type":"text","text":"请根据工单截图、客服通话和处理录屏，判断哪个质检结论更准确。"},{"type":"image_url","image_url":{"url":"tos://bucketname/image/ticket.png"}},{"type":"audio_url","audio_url":{"url":"tos://bucketname/audio/customer_call.wav"}},{"type":"video_url","video_url":{"url":"tos://bucketname/video/operation_record.mp4"}}]},{"role":"assistant","choices":[{"content":"客服准确解释了配送延迟原因，并在录屏中完成了补偿流程，结论完整。","score":1.0,"lm_loss_mask":1.0},{"content":"客服没有解决用户问题，建议直接判定为不合格。","score":0.2,"lm_loss_mask":0.0}]}],"thinking":{"type":"disabled"}}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58{
  "messages": [
    {
      "role": "system",
      "content": [
        {
          "type": "text",
          "text": "你是一名多模态质检助手，需要结合文本、图片、音频和视频判断回复质量。"
        }
      ]
    },
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "请根据工单截图、客服通话和处理录屏，判断哪个质检结论更准确。"
        },
        {
          "type": "image_url",
          "image_url": {
             "url": "tos://bucketname/image/ticket.png"
          }
        },
        {
          "type": "audio_url",
          "audio_url": {
             "url": "tos://bucketname/audio/customer_call.wav"
          }
        },
        {
          "type": "video_url",
          "video_url": {
             "url": "tos://bucketname/video/operation_record.mp4"
          }
        }
      ]
    },
    {
      "role": "assistant",            # 最后一个message的role必须为assistant
      "choices": [                    # 可以有多个response
        {
          "content": "客服准确解释了配送延迟原因，并在录屏中完成了补偿流程，结论完整。",
          "score": 1.0,               # 该response的得分
          "lm_loss_mask": 1.0         # 这个response是否计算 sft loss，默认为0.0
        },
        {
          "content": "客服没有解决用户问题，建议直接判定为不合格。",
          "score": 0.2,
          "lm_loss_mask": 0.0         # 这个response是否计算 sft loss，默认为0.0
        }
      ]
    }
  ],
  "thinking": {
    "type": "disabled"
  }
}

1{"messages": [{"role": "user","content": [{"type": "text","text": "\nLet's think step by step and output the final answer within \\boxed{}."},{"type": "image_url","image_url": {"url": "data:image/{image_extension};base64,{image_base64}"}},{"type":"video_url","video_url":{"url":"tos://bucketname/video/question.mp4"}}]}],"extra": {"answer": "A"},"thinking": {"type": "enabled"}}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29{
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "\nLet's think step by step and output the final answer within \\boxed{}."
        },
        {
          "type": "image_url",
          "image_url": {
              "url": "data:image/{image_extension};base64,{image_base64}"
          }
        },
        {
          "type": "video_url",
          "video_url": {
              "url": "tos://bucketname/video/question.mp4"
          }
        }
      ]
    }
  ],
  "extra": {
    "answer": "A"
  },
  "thinking": {"type": "enabled"}
}

1
2
3
4
5
6
7{"text": "火山方舟通过稳定可靠的安全互信方案，保障模型提供方的模型安全与模型使用者的信息安全，加速大模型能力渗透到千行百业，助力模型提供方和使用者实现商业新增长。"}
{"content":[{"type":"text","text":"支持运行超大规模的分布式任务，"},{"type":"text","text":"包含多种预置算法框架和自定义算法框架。"}]}
{"content":[{"type":"image_url","image_url":{"url":"tos://bucketname/image/figure.jpeg"},"loss_weight":0.0},{"type":"text","text":"学习这张图片对应的描述方式。","loss_weight":1.0}]}
{"content":[{"image_url":{"url": "data:image\/jpeg;base64,xx"}, "type": "image_url","loss_weight":0.0},{"type": "text", "text": "今天天气很好", "loss_weight": 1.0}]}
{"content":[{"type":"audio_url","audio_url":{"url":"tos://bucketname/audio/customer_call.wav"},"loss_weight":0.0},{"type":"text","text":"这是一段客服通话音频，可用于学习语音转写、说话人意图识别与问题摘要的表达方式。","loss_weight":1.0}]}
{"content":[{"type":"video_url","video_url":{"url":"tos://bucketname/video/demo.mp4"},"loss_weight":0.0},{"type":"text","text":"学习这段视频中画面与声音的联合描述方式。","loss_weight":1.0}]}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65{
    "text": "火山引擎机器学习平台是面向机器学习应用开发者，提供【开发机】和【自定义训练】等丰富建模工具、多框架高性能模型推理服务的企业级开发平台，支持从数据托管、代码开发、模型训练、模型部署的全生命周期工作流。"
}
{
    "content": [
        {
            "type": "text",
            "text": "支持运行超大规模的分布式任务，",
            "loss_weight": 1.0
        },
        {
            "type": "text",
            "text": "包含多种预置算法框架和自定义算法框架。",
            "loss_weight": 1.0
        }
    ]
}
{
    "content": [
        {
            "type": "image_url",
            "image_url": {
                "url": "tos://bucketname/image/figure.jpeg"
            },
            "loss_weight": 0.0
        },
        {
            "type": "text",
            "text": "学习该图片所对应的描述内容。",
            "loss_weight": 1.0
        }
    ]
}
{
    "content": [
        {
            "type": "audio_url",
            "audio_url": {
                "url": "tos://bucketname/audio/customer_call.wav"
            },
            "loss_weight": 0.0
        },
        {
            "type": "text",
            "text": "这是一段客服通话音频，可用于学习语音转写、说话人意图识别与问题摘要的表达方式。",
            "loss_weight": 1.0
        }
    ]
}
{
    "content": [
        {
            "type": "video_url",
            "video_url": {
                "url": "tos://bucketname/video/demo.mp4"
            },
            "loss_weight": 0.0
        },
        {
            "type": "text",
            "text": "学习这段视频中画面与声音的联合描述方式。",
            "loss_weight": 1.0
        }
    ]
}

1
2
3
4
5
6bucketname    #桶名称
├── sample  #桶内文件夹名称
    ├── data.jsonl
└── image   #桶内文件夹名称
    ├── xxx.png
    └── xxxx.png

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23{
  "messages": [
    {
      "role": "user",
      "content": [
         {
          "type": "image_url",
          "image_url": {
            "url": "tos://bucketname/image/xxx.png"
          }
        },
        {
          "type": "text",
          "text": "What’s in this image?"
        }
      ]
    },
    {
      "role": "assistant",
      "content": "It is the logo of Doubao LLM model."
    }
  ]
}

1
2{"messages":[{"role":"user","content":[{"type":"text","text":"What’s in this image?"},{"type":"image_url","image_url":{"url":"data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAMCAgICAgMCAgIDAwMDBAYEBAQEBAgGBgUGCQgKCgkICQkKDA8MCgsOCwkJDRENDg8QEBEQCgwSExIQEw8QEBD/2wBDAQMDAwQDBAgEBAgQCwkLEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBAQEBD/wAARCABAAEADAREAAhEBAxEB/8QAHQAAAgEFAQEAAAAAAAAAAAAAAAgGAQMEBwkFAv/EADgQAAEDAwIDBAcFCQAAAAAAAAECAwQABQYHERIhMQgTQVEJFDJhcYGRFRYiQlMzNlJydIKSobP/xAAcAQABBAMBAAAAAAAAAAAAAAAABAUGBwEDCAL/xAA0EQABAwMCAwUFCAMAAAAAAAABAAIDBAURBiESMUEHUWFxoRMyUpGxFBUiIzSBwdFCcvD/2gAMAwEAAhEDEQA/AOqdCEUIVCQBuTsBQhYZvdlDvcG7wu86cHrCOL6b1u+zTY4uA48ivXA7uWYCCAQQQehFaV5VaEIoQihCKELz79fbZjNnl368SUsQ4TRddWfADwHmT0A8zSilpZa2ZtPCMuccBemMMjg1vNJXqjr3l+oM56PDmv2uyhRSzEYWUlafNxQ5qJ8ugq9bBo+itEYfM0Pl6k8h5BSKnoo4Rk7lax4lcXHxHi89+dS7gbjhxsleAp/gnaDzHStC5TsiRd7NHQXH7e84VHgHM90o+yrboOh8ajF40ZRX3aMCOU8nDlnxHcm64wRiF0oG4GU6On+f4tqdicDM8Nubc62XBvjQtJ/EhX5kLT1StJ5EGqMutqq7LVvoq1nC9p3H8jvB6FRyGZlQwSRnIKkVNy2ooQihCXPthZRIi2my4lHcUlE5xcuQAfaS3sEA/wBxJ+QqzOze3tlqJaxw9wYHmefona1xguLz0SsVcKel9NtuOrDbTalrV0Skbk/IV5e9sY4nHAWCQOa9ZzT/ADSfAdLWH3l1l1tSeJMFwggj4Ug+/LdFIA6doIPxBaJpI3RuaSNwtPdnjtB5t2Zs1cTJhzHsdmP93drS8lSOIA7d62Fey4kfUcj7pfrHSNt13Qh8D2+3aPwuBB/Y46H0VNUF0FHMWscC3O4yur2DZxjGo+Lwcxw+6tXC13BsONOoPMHxQodUqB5EHmDXIdztlVZ6p9FWMLZGnBB+o7wehU4hmZOwSRnIK96kC2ooQlR7ZH70Y9/QO/8ASrg7M/00/wDsPonu1e47zS/WCJKyrN7Lp7YkB+8Xp7hQj8rDI5uPueSUpBPvPIVY1wqG223y3GbZjB8yeTR4krNfdI6NzYhu93IfyV0BwbSzC9P4LUax2dj1hKAHZjiAp51W3MlR5jfyHKucbpfa67yF9Q846DoP2TVLUSTHLipaehpnCTu5JIsrtFqulyuMS5W+NMYMl0cD7SVgjjPmKlNHX1VEQ+nkLT4EhcdXKolprjMYXEYe7kfEq5o68NFcjcm4e47HslxWDcrMVlUZw/qtA/s3B5jkRyI6EOd4vk2oIAy5Djkb7r/8vI9489wplpntErbVKI6v8yM8+8eKdaO+3JYbktHdDqAtJ9xG4qCEYOF0jFI2VgkbyIz81crC9pPO3ff4uMTbPeJfMM253gR4rWXNkpHxNXV2TUj61s0LOrh8sJbHXMt9JJO/p6lQb0a9idybMM51SvI76XHaYtrDh6I70lawny5IQPhT7221QoaajtEOzTl58cbDPqolaZZK2eSrmOSU/lc8KQK3JdSxHdfWQEtoUok+AA3rI3K1zPEcbnnoCkomO+sTH3/1HVr+pJp+bsFxdWSe2qJJO8k+qy8fs8m/3uFZ4iCpyW8lsbDoN+Z+AG5ry9wY0kpVZ7fJdK+KkiG7iB/ZTmR2URo7UdsbJaQlCfgBtTGTk5XY8UYijbGOgAVysLYuePpQrhJGXYTawohg22Q+R/ErvQB9Bv8AWuk+weFhpauU8+Jo9FGtQzvDWQDlzUk9F3kME2bOcVK0iamVFuCU781NFCkE/Igf5Cmrt3pJBV0lXj8Ja5v7g5+izp144Hs6809VUCpIolqpfkY/g1zlcYS6+0YzI35la+X+huflW6nZxyAKK60ubbVZJ5SdyOEeZ2SnMsuyHUR47SnHHCEoQkblRPQAU8kgblcoRRSVDxHGMuPIBMdo7pcvFGPvBfGx9qyEbIb6+roPh/MfHy6U2VNR7Q8LeS6M0Bov7ij+3Vg/OcNh8I/s9VtCkas1FCEqnb60BverGD2/McPgrmXvFe9UuI2N3JMRexWEDxUkpCgPEcXjVtdkuroNO3F9JWu4YpsDPQOHInwPJMt6onVMQezm36LnfpVqnmeimbx8zxCR6vcInEy+w8k92+2T+Nl1PI7Hb4ggEcxXTOoNP0Gq7eaOr3YdwRzB6EH/ALKilNUyUcnGzmnMielHgfZqBO0hkquHAAsM3RIZK/dujiA38OdUZL2ETiU8FY3g7y05x88KQDUTQ3Lmb+a2Jit21v7RVpiXfJsai2CA8538WKkLShhsjYKcWrm4sjnsAAN+nWqwv9Fa7JVmkoJTJw7FxxueuAOgVbaggvOvKltNSs4adh947Anv8ceC3ngWk+P4SlMsj165bfikuJ9jzCE/l+PWotNUOl26Kc6Y0Pb9OASAccvxHp5DopxSdTVFCEUIRQhao1C7LOg+p9wXd8s0+gruDp3clxFLiuuHzWWiniPvO5qW2fXN/sUfsqKpcGDocOA8gc4SKe3U1QeJ7d1h4T2Q+zzgM9F1senEJ2Y0Qpt6e4uWUEHcFIdKgD79t63XTtB1HeIzFU1TuE8w3Dc+fDheYrXSwnLWfPdbiSlKEhCEhKQNgANgKhhOdylzWhow0bKtCyihCKEL/9k="}}]},{"role":"assistant","content":"It is the logo of Doubao LLM model."}]}
{"messages":[{"role":"user","content":[{"type":"text","text":"What’s in this image?"},{"type":"image_url","image_url":{"url":"data:image/jpeg;base64,/9j/4AAQSkZJRgABAQAAAQABAAD/2wBDAAIBAQEBAQIBAQECAgICAgQDAgICAgUEBAMEBgUGBgYFBgYGBwkIBgcJBwYGCAsICQoKCgoKBggLDAsKDAkKCgr/2wBDAQICAgICAgUDAwUKBwYHCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgoKCgr/wgARCABQAFADAREAAhEBAxEB/8QAHAAAAgIDAQEAAAAAAAAAAAAACAkGCgAFBwQD/8QAGwEAAgIDAQAAAAAAAAAAAAAAAwUEBgACBwH/2gAMAwEAAhADEAAAAK/+ZnuNucrW6OVuCLK99IyTRPCJop9QxzzM9yyfaETJWa9HNJuYgpXcfKEnma109vqQcRZFbGrvGwN11iC0IAWrNgS7ULjvBnPMMgbSgkdhrb67xSq9lafN6dKyZlxkR8+vIZK3bfVb9ohYXIdDI4bIrBPUOYjVGkEzLi+COatdzu/eCI3P4ezXITNYciFHjgbX1bk3330zM1cc1aLnfQCkRXNiRYxUDxb5/CvFt3Lp/J9k3TDNDlEocFavnHQenLnby4TOJQncLnpNZfudFpaamM0SSoRI0sKWZENy2fCRl8HOupSMEsfb7zAz3afs0uJXfq9gU+oYWT7OiY81WroRPTY5n1b6Zuuu/wDMyPeIuQANWxrDzPMz3CKkBjwiMNgtScBKTqxS853Fz3XbPM//xAAoEAAABgMAAgIBBAMAAAAAAAADBAUGBwgBAgkAChEhEBITFRYmMTL/2gAIAQEAAQwA8xj585m+rLM8/JQcq3/XVuLW9WHkRzUpQzUzDBrI0DB+Tux3OaKFgVAWbGElI5XLpNSi1S5q04anVOOLVquPHOG4LaWkqUasNVPV+lPq22JgM+fkyhA56TmN+PjPz8ecA+JTKpFGiLbufEPClMzhcCQ1kQ04l05qXJ9T+ij/ALYSEei5uOkwVZWrLwMDnbQHO3ihhQaKiEtIiiOUOVa68XKrU8zL9MvMV5EqL3qh6+kR4kuMB9yh3tpwDjO7jIPz7URkobUmb4+/jPnq181D892KFvtJiCQMsbztvag3EUFjR22FLICk1iwYwwho8NjQBlNWXbIvYGMoKZpxQGgb1nJSereCXp1sKVb4/QLmfONGFL/NG5qqNLgjLKnFt/05ohL+QU7z2meaCVAM0Eb/AMVFf2W7x4qo2qe84osi5KZh5EV85xjGds5+uxs5DypZtVRgDWdyaqtBZFINTA4gWOSci0QgRFSmWO88JDl2Pg6lPrOMYsC+YCVUA7FU2uRtZI2+rUrc0rsNR/xmfHMtUmaBPFAjpff9QfYWqjbuFzilSLVVmHltXxj4x8eOo7/GNZSUfLLuY055YcK2bE/WJBBmV2WrmpZYTITT4XMbqtINqJDO1zszXxsb5Nqg4Tcyf2znGL03/wCd0Y2a2YL+roWf74n+H4PtlVEqQrszldsHUwkGmpwCcDn508x948exbJ1lrBPX/c5ltyUhrBTfHxnlTELXndmO2IHiVwYTOb/J+slGVNVX49Q1FQPOtxNkc0Gy9DAWTF5+NKRO9oEuXsTKrF2kWaKC6R0RHNIxYPDS/d/gw9dxP16+ceLVtu4XOGLJRSnodW1c8FgYgODnH1aUroXnJxk9P+eZltWBXqxJtjSqvhoZOM3tlyI2m6UoA4HMrdsv7yHlWS0XCfcO5cOVfjg0/pjeRUkDybk85bythmzz/QgNBddddNMaa4+MdhrVtqnvOGVJRVXmeRFf1Y+l6VAM0n6ASoP+03vJ15L0QsGqmnK8oZwSV2fwd5woSIYSnfFam7jIUAINT3cNE9URz6MgtmQ7CS+6QYdXnpht5mf1wy09PQd9ybfJ4Lh+hFPk2ilakmt6Q/DDkLee0j0qPz1YrShMZL5EyxsfXnEnv3Gl3WSRgO3j2Q2rMzxerPjxtm3i/XQnoqVZr2PqxtKUEWHK4i5cOkfRMEok/wC1GVTVTFdEKC5OBLKMZ2KHrNewjONILTuqtcoQC3n0V5t9ua9dEXuaiEozFNkPPv121ZNI4zXKiQEu4U5m/GPryXuolyrNtZusK0M5rTvSWc6mYqBhbgLBYMSkHci19LWSTi0cJKfDRmD2KrHaV5TpmZVUGqlFZ2sEblF+r0sSk9NVhxkbFP8AYzlJuyG3SrNdW/H/xAA5EAACAgECBAEICQIHAAAAAAABAgMEBQARBhITITEQFCIyQUJRgQcVFjNSYWJxgiPBJEODkZKh0f/aAAgBAQANPwDyS9J8ZwZikrHP5OCWqZBNK79RMXySPCDDNDJOSk8ckVchHPDtmDKDjvjXGwZPMJdgjiAv+e2kJquGgWblr9GGOQu8cceoG5ZU4dx1nIRj/VgjaI/JtOCYcJkYJqNubbuenHYVDL/Dm1m71jIWeM+EcTDic2MhMkwa61uuitZk55nlKWetC8gV5I5NVqCTS8NZK5D9qKRjgmeyQiRQw3494QYxABZZrKQrXk5DM/l4qwcdqjVyVCSH7D0rMIJpxxTorpfaNylmVlDR7vWj2QTPZpwmSeVvYB8B7SfAAdySBrGWjFDg6FkrWndT97YZD/iZPy+7T3ebu5/IbDVKdZq1mrO0csMindXRlIKsCAQQdxrMukmbwnFN6edLGx7vGxfeGUj3h81OqTrBxDw9bcGzjLBG4VtvXRvFJB2YA+BBAxgsXLNGhBDRpccc8kk80VsgKiX2kd3juv67OYrDFCk1bX0Y3pqnDcNmaCY3uKOjC6E13ikPTqQWBYEu8TLZNNomfpTBNZgCDnifZl5weY/xT/uUHUXpzO2pXChaybKo/HI/go1OAxx2JwptlP3d5E1LOYcRxvh4282n/AlhO5rTEe43YkHkZ9cY4uzjrlN39C0wXqxfzV0BH5FvJ9KObNHjPFw1asEGMz4r86TRCMo8nnscFmeQmNiJ4J5HlJsogyPCtTP8Z1sxjkrZA5u/Cli2lpRFG3UhdxVUSgyJDVhiYkx6GsM5giHs5z3b+w1esDrGGF5H5fyVAWPyGssURH4iwdugbMp9iy2IkQ9/1a9ms7UaCzhs/fgRbcTbgjkkYFvl4HUmcr57grJGQuYxBOjvUd/fKdhv7ySKTqaNXjPxBG41j+Fbef4Mr4fHJZyAzdCGSxUSqpikYSTOhqsIgJHhtTRKQZPJXoTSf8UY6s5OaRnJ+Lk6oGUHIZKZ2Su0MfV5OigBkZ9uRQHXcnuRrGIfOpcTEXFJo7UdbexWmaQbCWSJWZHJQtzcpQO6cusrbhx2Rv28bWNavLyhUgNi2eUMAUB5ByLzemwOovpLxmNj4Uy1WatFh8o98UbEapLvHEAs7mQQMY90BPpDVeBI1P5KNv7eWbFzoPnGw0t+VT8nOsm7x363gZYmBRgD4rupK8w7gE7EazFXzae5n54ZTHAWRmiQRxRgKzRox3BJKDUlgBoQe4XlJ31UzcWWtcBTQNJVFwQ14JpoG6gWNpYqlcMShO8fiRsBjrbZGaKMf0zbdpJJJT4bkySu/ceJB8RpSRC59qAkDyY/hWpgOM7OYyKWcgM3QhSvbe2wlkbqTOgtKZSJHhtQysAZNNCwP+x0mXmABH6zrJzJ9T5+d+WvXlb/ACbJ9yJz3Evgjet6JLJJXWSvIWEiOGG6uCDs6kEEEHYg9jpZiJZ/qVRGI+/pBxaLk/IahRhBWQgz3ZgCejXj33kc/DwHixA1xFxZf+oscfSXHY6ApBFB+tuaOSRmPi0p22UAAeAGshwrbwHBlnD5FK2QObvwyVqj1WMsbdSF3NpjExkSGrNKoJj19KWbW9wZk5bVWCDGZ8VyjwymQI8nnscFaCMCRiJ4II0iJsu41ckMk2WwOVsU5S58SUR+kT+6HU1tpYstxDxDYS1ChVQIQajQIUXYkboW79ydYyCOZcVYyU1yOKeT0nC9dmZARyHYEayUE6QZujRWaSJ0idwQkpKn1D476n7GfM8OwzlE/AgWdFjX4KqhdYm3aniylmgtZ360zSlSgZh2LEePk+jG9Bb4klqwQTG7xR0ZkcCykkh6dWCya5i2iZbJtrKrmKEprG+b06t6/PDRpccc8iQRSVQSqJfaR0SSknrs4krqULw1qMZkuZHKW0gghUe1ncgDVzP1aef47uQOmNxdVpwk0sUfaSyVQk77KntHPrKAWnyIdWFjnHMHBXtykEbbdttttVJVlq2YQOaNwex/9HgQSDrhqzAkWYxGXlxlmVJYI5h1QUmjMiiTlPKqAkaiptbpYXL3I54sjEn3nQmULzug7lCoPL3G4DbcVYOSrdtY2/JD9h6VmEgXZJoHV0vtG4etErBo/QsyEJ0Us+XhigKmJNrkWcIERA0zIE87lITvPPzzuXJeRtO6qIJ2CPzbj0QCe57gdgRvvsTqgCuOxnEJkFmlEPCKCxGd1T4K6uFHhrP3J6GMv3+LpL/Rni3BL10hhPsJAL/v2ZN85lJbeVsB1aSWwxJI5E7IB6oXYKoAGsfKZKWexN5q12ByhUtFJEQ0J7sOZW3+BHl//8QAJBEAAQQCAgMBAAMBAAAAAAAAAQACAwQFESExEBITQQYUIlH/2gAIAQIBAT8AQG+lTxT5f9ScBRVK1Zu0/LUov8lygyNSc+rXKWlXnB2FaxUkeyzkIgjwASsbQbE36O7Uj2wsLnfiyOUkneWg8IyOJUcrmnYUGVnhk3tUrsd2PY7V7HNlaXxjldbCxNUySfR3Q8Zu2QPm0o7KdIyMcoTu70o5dlYawYrIG+12Flqnzf8ARv6qUIggA0nHTSVfmMsxKc71aSmiR7y562EDzwsdOfoP+qM+0YKuwiaAjxaOoHKY7cV8W2XehOlLR+bdtO9IdKKrLKNqpC9loNco+IwtceLQ3A5TD/RViaSGfbCmWpnAg/q9udJt98f5tPmklf7fqwbpnY9pk78UZhNAHbUw3EQrHEhU1USt9h2vUs209of2/rz0mROeVhMVUdWEjxsprQ0aars3xrk75WJt/N/zd0VoEKXEU5dkjlR4WkxuiNr+RNZXvmOH8WEiF+8IZSm/xuEDQcqVUVIBHvxlbRkk+Y6CB0VQyLJW+kh5TpGMbtxWQy5ZCRDyVL9Z5i+TklQukqyiSM6IWPzk8kAMg2VVyDLB0eCsjfbEwxs7RJJ8A6TrU0g08rgqajHIdjhDGfpKjYyJnqEJHNOwiSfH/8QAKBEAAQQCAgIBAwUBAAAAAAAAAQACAwQFERIhEDETBiJBFSAjMlGx/9oACAEDAQE/APE+Rjj6Z2U6zYnJAKjxdx42GqahbgG3DpR27EWu1XvxydP6P7Ltxz3FjPX/AFRsMj+IWMxscLORHaEbQpI2vGlPioZo9elbpvqSaKp3iw8JPXi/YEcfAez4w1UPfzKZpoUNeWc/b6RqRM9nZUkBA2sxAH1yf88Y6wZY+B9hXJfkmJ30mdnQWNiEcAULA+QApzo2sDI18fe0GjWlk6vGM/4U8aeQqUhimB34rjcoVcajCjsvow/KG72quT+eTThraAU92CE6VqaKai57U/fM+a/UoVf+gWPr17FDUg2p6FSLtn4UcZ47TcNDY+7lpMrQ12GP8BZ1sDci8RevFyIxTkKLp4VTuIKrkXV/4z6THsfp3sI/pQq/b/ZPstgBO1m8xdFl0bHaCcS4klUovkmA/CyFf5I+Y9hDpQ5O3D0D0pMvbe7YK+lf58aJJ+9rPltDHOnh7ITvqGZ525quWTbmMhGvGPriOPmfZ8W6TmHkwdJrXOOgFjcYx87TP6UBiggEcQ0ApPjsQGKTsFZPCwQ2nNjOgrFJ0A37VGmXnm71+xteJhJaENhVspLEOLuwjnN7aGqWV8zy4osaRo+f/9k="}}]},{"role":"assistant","content":"It is the logo of Doubao APP."}]}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23{
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "image_url",
          "image_url": {
            "url": "data:image/{image_extension};base64,{image_base64}"
          }
        },
        {
          "type": "text",
          "text": "What’s in this image?"
        }
      ]
    },
    {
      "role": "assistant",
      "content": "It is a cat."
    }
  ]
}

1
2
3
4
5
6
7
8import base64

# 将结果填入 url 字段内
def encode_image(image_path):
    extension = image_path.split(".")[-1]
    with open(image_path, "rb") as image_file:
        base64_image = base64.b64encode(image_file.read()).decode('utf-8')
        return f"data:image/{extension};base64,{base64_image}"

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23{
    "messages": [
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": "file:./image/doubao_logo.jpeg"
                    }
                },
                {
                    "type": "text",
                    "text": "What’s in this image?"
                }
            ]
        },
        {
            "role": "assistant",
            "content": "It is the logo of Doubao LLM model."
        }
    ]
}

1{"messages": [{"role": "user","content": [{"type": "text","text": "你觉得这个恐怖吗？"},{"type": "text","text": "[0.0 second]"},{"type": "image_url","image_url": {"url": "base64/相对路径"}},{"type": "text","text": "[1.0 second]"},{"type": "image_url","image_url": {"url": "base64/相对路径"}},{"type": "text","text": "[2.0 second]"},{"type": "image_url","image_url": {"url": "base64/相对路径"}}]}]}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43{
  "messages": [
    {
      "role": "user",
      "content": [
        {
          "type": "text",
          "text": "你觉得这个恐怖吗？"
        },
        {
          "type": "text",
          "text": "[0.0 second]"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "base64/相对路径"
          }
        },
        {
          "type": "text",
          "text": "[1.0 second]"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "base64/相对路径"
          }
        },
        {
          "type": "text",
          "text": "[2.0 second]"
        },
        {
          "type": "image_url",
          "image_url": {
            "url": "base64/相对路径"
          }
        }
      ]
    }
  ]
}

1
2
3
4
5
6{
  "type": "audio_url",
  "audio_url": {
    "url": "data:audio/wav;base64,<BASE64_ENCODING>"
  }
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15{
  "content": [
    {
      "type": "video_url",
      "fps": 1,
      "video_url": {
        "url": "tos://bucket/video/demo.mp4"
      }
    },
    {
      "type": "text",
      "text": "请你描述下视频中的人物的一系列动作。"
    }
  ]
}

1
2pip install opencv-contrib-python
pip install arkruntime

1
2
3
4
5
6...
# 1. CONSTANT_INTERVAL：固定时间间隔抽帧，可以调整时间间隔
# 2. EVEN_INTERVAL：固定总帧数均匀抽帧，可以调整总帧数
        extraction_strategy=Strategy.CONSTANT_INTERVAL,
        # extraction_strategy=Strategy.EVEN_INTERVAL,
...

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248from typing import Optional
# Import the OpenCV library for video processing
import cv2
# Import the time library for timing
import time
# Import the file operation library for directory deletion
import shutil
# Import the enum library for defining frame extraction strategies
from enum import Enum

# Import the Base64 encoding library for converting images to Base64 strings
import base64
# Import the operating system library for file path operations
import os
# Install SDK: python -m pip install --upgrade arkruntime
from arkruntime import Ark
from arkruntime._exceptions import ArkAPIError


# Define the frame extraction strategy enumeration class
class Strategy(Enum):
    # Fixed interval frame extraction strategy, e.g., extract one frame every 1 second
    CONSTANT_INTERVAL = "constant_interval"
    # Even interval frame extraction strategy, extract frames evenly from the entire video based on the set maximum number of frames
    EVEN_INTERVAL = "even_interval"

def preprocess_video(
        video_file_path: str,
        output_dir: str,
        extraction_strategy: Optional[Strategy] = Strategy.EVEN_INTERVAL,
        interval_in_seconds: Optional[float] = 1,
        max_frames: Optional[int] = 10,
        keyframe_naming_template: str = "frame_{:04d}.jpg",
) -> list[str]:
    """Extract frames from video according to the specified strategy
    Args:
        video_file_path (str): Path to the video file
        output_dir (str): Output directory
        extraction_strategy (Optional[Strategy], optional): Frame extraction strategy.
             Constant interval (e.g., extract one frame every 1 second) or
             Even interval (extract frames evenly from the entire video based on the set maximum number of frames)
             Defaults to constant interval (1 frame per second).
        interval_in_seconds (Optional[float], optional): Time interval for constant interval frame extraction. Defaults to 1 second per frame.
        max_frames (Optional[int], optional): Maximum number of frames to extract. Defaults to 10 frames.
        keyframe_naming_template (_type_, optional): Naming template for extracted key frame images
    Returns:
        list[str]: List of paths to the extracted key frame images
    """
    # Check if the output directory exists; create it if it does not
    if not os.path.exists(output_dir):
        os.makedirs(output_dir)
    # Open the video file using OpenCV
    cap = cv2.VideoCapture(video_file_path)
    # Get the frame rate of the video
    fps = cap.get(cv2.CAP_PROP_FPS)
    # Get the total number of frames in the video
    length = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))

    # Select the frame interval based on the strategy
    if extraction_strategy == Strategy.CONSTANT_INTERVAL:
        # Calculate the frame interval for constant interval extraction
        frame_interval = int(fps * interval_in_seconds)
    elif extraction_strategy == Strategy.EVEN_INTERVAL:
        # Calculate the frame interval for even interval extraction
        frame_interval = int(length / max_frames)
    else:
        # Throw an exception if the strategy is invalid
        raise ValueError("Invalid extraction strategy")
    # Initialize the frame counter
    frame_count = 0
    # Initialize the list of key frames
    keyframes = []
    # Loop to read video frames
    while True:
        # Read one frame
        ret, frame = cap.read()
        # Exit the loop if frame reading fails
        if not ret:
            break
        # Check if the current frame is a key frame
        if frame_count % frame_interval == 0:
            # Generate the filename for the key frame
            image_path = os.path.join(
                output_dir, keyframe_naming_template.format(len(keyframes))
            )
            # Save the key frame as an image
            cv2.imwrite(
                image_path,
                frame,
            )
            # Add the key frame path to the list
            keyframes.append(image_path)
        # Increment the frame counter
        frame_count += 1
        # Exit the loop if the maximum number of key frames is reached
        if len(keyframes) >= max_frames:
            break

    print("[Video Frame Extraction Completed]")
    print("[Number of Extracted Frames]", len(keyframes))
    # Return the list of key frame paths
    return keyframes

def resize(image):
    """
    Resize the image to fit the specified dimensions.
    Args:
        image (numpy.ndarray): Input image in numpy array format.
    Returns:
        numpy.ndarray: Resized image.
    """
    # Get the original height and width of the image
    height, width = image.shape[:2]
    # Determine the target dimensions based on the image's aspect ratio
    if height < width:
        target_height, target_width = 480, 640
    else:
        target_height, target_width = 640, 480
    # Return the original image if its dimensions are already less than or equal to the target dimensions
    if height <= target_height and width <= target_width:
        return image
    # Calculate the new height and width while maintaining the image's aspect ratio
    if height / target_height < width / target_width:
        new_width = target_width
        new_height = int(height * (new_width / width))
    else:
        new_height = target_height
        new_width = int(width * (new_height / height))
    # Resize the image
    return cv2.resize(image, (new_width, new_height))

# Define a method to resize the image at the specified path to an appropriate size and convert it to Base64 encoding
def encode_image(image_path: str) -> str:
    """
    Encode the image at the specified path
    Args:
        image_path (str): Path to the image file
    Returns:
        str: Encoded image string
    """
    # Read the image
    image = cv2.imread(image_path)
    # Resize the image
    image_resized = resize(image)
    # Encode the image in JPEG format
    _, encoded_image = cv2.imencode(".jpg", image_resized)
    # Convert the encoded image to a Base64 string
    return base64.b64encode(encoded_image).decode("utf-8")

def construct_messages(image_paths: list[str], prompt: str) -> list[dict]:
    """
    Construct a list of messages containing text and images.
    Args:
        image_paths (list[str]): List of image file paths.
        prompt (str): Text prompt.
    Returns:
        list[dict]: List of messages containing text and images.
    """
    print("[Assembling Request Parameters]")
    # Initialize the content list with a text message
    content = [
        {
            "type": "text",
            "text": prompt,
        },
    ]
    # Iterate through the list of image paths
    for image_path in image_paths:
        # Construct an image URL message for each image path
        content.append(
            {
                "type": "image_url",
                "image_url": {
                    # Convert the image to a data URL using Base64 encoding
                    "url": f"data:image/jpeg;base64,{encode_image(image_path)}",
                    # Specify the image detail level as low
                    "detail":"low"
                },
            }
        )
    # Return the list of messages containing text and images
    return [
        {
            "role": "user",
            "content": content,
        }
    ]

if __name__ == "__main__":
    # Replace with the path to your video file
    video_path = "<VIDEO_FILE_PATH>"
    # API Key retrieved from environment variables; configure your API Key in advance
    ark_api_key = os.environ.get("ARK_API_KEY")
    # Replace <MODEL> with the Model ID of your model
    model_id = "<MODEL>"
    # Replace with your text prompt; this is for example use only
    prompt = "Describe the video content"
    since = time.time()
    # Delete historical frames
    if os.path.exists("videoFrames"):
        shutil.rmtree("videoFrames")

    selected_images = preprocess_video(
        video_file_path=video_path,
        output_dir="videoFrames",

        # Two frame extraction strategies
        # 1. CONSTANT_INTERVAL: Extract frames at a fixed time interval; adjust the interval as needed
        # 2. EVEN_INTERVAL: Extract a fixed total number of frames evenly; adjust the total number as needed
        extraction_strategy=Strategy.CONSTANT_INTERVAL,
        # extraction_strategy=Strategy.EVEN_INTERVAL,

        # Frame interval for constant interval extraction strategy
        interval_in_seconds=1,

        # Maximum number of frames to extract from the video
        max_frames=20
    )

    print("[Frame Image Compression Completed]")
    preprocess_time_cost = time.time() - since

    client = Ark(
        api_key=ark_api_key,
        base_url="https://ark.cn-beijing.volces.com/api/v3",
    )
    since = time.time()
    try:
        response = client.chat.completions.create(
            model=model_id,
            messages=construct_messages(
                selected_images,
                prompt,
            )
        )

        print("[Video Understanding Result]\n", response.choices[0].message.content)
        print("\n[Total Tokens Consumed]", response.usage.total_tokens)
        print("- Input Tokens:", response.usage.prompt_tokens)
        print("- Output Tokens:", response.usage.completion_tokens)

    except ArkAPIError as e:
        print(e)

    api_time_cost = time.time() - since
    print("\n[End-to-End Total Time Consumption]", round(preprocess_time_cost + api_time_cost, 2))
    print("- Video Preprocessing Time:", round(preprocess_time_cost, 2))
    print("- Result Generation Time:", round(api_time_cost, 2))

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179{
  "messages": [
    {
      "role": "system",
      "content": "你是豆包"
    },
    {
      "role": "user",
      "content": "把北京上海的天气发给Alan"
    },
    {
      "role": "assistant",
      "content": "分别查询北京和上海的天气。",
      "tool_calls": [
        {
          "type": "function",
          "function": {
            "name": "GetCurrentWeather",
            "arguments": "{\"location\": \"北京\"}"
          }
        },
        {
          "type": "function",
          "function": {
            "name": "GetCurrentWeather",
            "arguments": "{\"location\": \"上海\"}"
          }
        }
      ],
      "loss_weight": 1.0
    },
    {
      "role": "tool",
      "content": "北京天气晴，18~25°，微风"
    },
    {
      "role": "tool",
      "content": "上海天气阴，20~30°，大风"
    },
    {
      "role": "assistant",
      "content": "已经查到北京上海天气，需要发送给alan",
      "tool_calls": [
        {
          "type": "function",
          "function": {
            "name": "SendMessage",
            "arguments": "{\"receiver\": \"Alan\", \"content\": \"今天北京天气晴，18~25°，微风；上海天气阴，20~30°，大风\"}"
          }
        }
      ],
      "loss_weight": 1.0
    },
    {
      "role": "tool",
      "content": "发送成功"
    },
    {
      "role": "assistant",
      "content": "已经查到北京上海今日天气，发送给了Alan",
      "loss_weight": 1.0
    }
  ],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "MusicPlayer",
        "description": "歌曲查询Plugin，当用户需要搜索某个歌手或者歌曲时使用此plugin，给定歌手，歌名等特征返回相关音乐",
        "parameters": {
          "properties": {
            "artist": {
              "description": "表示歌手名字",
              "type": "string"
            },
            "description": {
              "description": "表示描述信息",
              "type": "string"
            },
            "song_name": {
              "description": "表示歌曲名字",
              "type": "string"
            }
          },
          "required": [],
          "type": "object"
        }
      }
    },
    {
      "type": "function",
      "function": {
        "name": "GetCurrentWeather",
        "description": "查询当前的天气",
        "parameters": {
          "type": "object",
          "properties": {
            "location": {
              "type": "string",
              "description": "地理位置，比如北京市"
            },
            "unit": {
              "type": "string",
              "description": "温度单位",
              "enum": [
                "celsius",
                "fahrenheit"
              ]
            }
          },
          "required": [
            "location"
          ]
        }
      }
    },
    {
      "type": "function",
      "function": {
        "name": "LinkReaderPlugin",
        "description": "当需要解析网页内容时，调用本插件",
        "parameters": {
          "type": "object",
          "properties": {
            "url": {
              "type": "string",
              "description": "需要解析网页链接,最多3个；"
            }
          },
          "required": [
            "url"
          ]
        }
      }
    },
    {
      "type": "function",
      "function": {
        "name": "WebSearchPlugin",
        "description": "当需要搜索互联网内容时，调用本插件",
        "parameters": {
          "type": "object",
          "properties": {
            "keywords": {
              "type": "string",
              "description": "需要搜索的内容"
            }
          },
          "required": [
            "keywords"
          ]
        }
      }
    },
    {
      "type": "function",
      "function": {
        "name": "SendMessage",
        "description": "需要发送微信消息时，使用本函数",
        "parameters": {
          "type": "object",
          "properties": {
            "receiver": {
              "type": "string",
              "description": "接受对象"
            },
            "content": {
              "type": "string",
              "description": "需要发送的内容"
            }
          },
          "required": [
            "receiver"
          ]
        }
      }
    }
  ]
}

1{"messages":[{"role":"system","content":"你是豆包"},{"role":"user","content":"把北京的天气发给Alan"},{"role":"assistant","content":"","tool_calls":[{"type":"function","function":{"name":"GetCurrentWeather","arguments":"{\"location\": \"北京\"}"}}],"loss_weight":1.0},{"role":"tool","content":"北京天气晴，18~25°，微风"},{"role":"assistant","content":"已经查到北京天气，需要发送给alan","tool_calls":[{"type":"function","function":{"name":"SendMessage","arguments":"{\"receiver\": \"Alan\", \"content\": \"今天北京天气晴，18~25°，微风\"}"}}],"loss_weight":1.0},{"role":"tool","content":"发送成功"},{"role":"assistant","content":"已经查到北京今日天气，发送给了Alan","loss_weight":1.0}],"tools":[{"type":"function","function":{"name":"MusicPlayer","description":"歌曲查询Plugin，当用户需要搜索某个歌手或者歌曲时使用此plugin，给定歌手，歌名等特征返回相关音乐","parameters":{"properties":{"artist":{"description":"表示歌手名字","type":"string"},"description":{"description":"表示描述信息","type":"string"},"song_name":{"description":"表示歌曲名字","type":"string"}},"required":[],"type":"object"}}},{"type":"function","function":{"name":"GetCurrentWeather","description":"查询当前的天气","parameters":{"type":"object","properties":{"location":{"type":"string","description":"地理位置，比如北京市"},"unit":{"type":"string","description":"温度单位","enum":["celsius","fahrenheit"]}},"required":["location"]}}},{"type":"function","function":{"name":"LinkReaderPlugin","description":"当需要解析网页内容时，调用本插件","parameters":{"type":"object","properties":{"url":{"type":"string","description":"需要解析网页链接,最多3个；"}},"required":["url"]}}},{"type":"function","function":{"name":"WebSearchPlugin","description":"当需要搜索互联网内容时，调用本插件","parameters":{"type":"object","properties":{"keywords":{"type":"string","description":"需要搜索的内容"}},"required":["keywords"]}}},{"type":"function","function":{"name":"SendMessage","description":"需要发送微信消息时，使用本函数","parameters":{"type":"object","properties":{"receiver":{"type":"string","description":"接受对象"},"content":{"type":"string","description":"需要发送的内容"}},"required":["receiver"]}}}]}