Repository navigation
feat(provider): support native video input modality - #10202
JosephTian876 wants to merge 6 commits into
Conversation
Add ProviderRequest.video_urls so providers that declare the �ideo modality receive the video itself instead of only a text placeholder. Fixes two failure modes that a video block would otherwise introduce: - gemini_source._prepare_conversation dispatched user content with a catch-all �lse that read item["audio_url"]. A video_url block raised KeyError outside the try that guards the API call, and because the block was persisted the error repeated on every later request. The branch is now exhaustive and unknown block types degrade to text. - assemble_context output is replayed from history on every request, so video blocks are now flagged _no_save and dropped when the message is written back. Video is opt-in: an empty modalities list keeps its existing "supports everything" meaning for text/image/audio but not for video, so existing configurations are unaffected.
![sourcery-ai[bot]](https://avatars.githubusercontent.com/in/48477?s=60&v=4){kind=small}
There was a problem hiding this comment.
Hey - I've found 2 issues
Prompt for AI Agents
Please address the comments from this code review:
## Individual Comments
### Comment 1
<location path="astrbot/core/agent/runners/tool_loop_agent_runner.py" line_range="344-352" />
<code_context>
) -> dict[str, T.Any]:
modalities = self.provider.provider_config.get("modalities", None)
if not modalities: # Unconfigured (None or empty list) defaults to support all modalities for backward compatibility
- return await request.assemble_context()
+ # Video is excluded from that default. It postdates the "supports
+ # everything" convention and is not universally supported, so a
+ # provider that never opted in would fail the whole request rather
+ # than degrade. Video is therefore strictly opt-in via the
+ # `modalities` list.
+ if not request.video_urls:
+ return await request.assemble_context()
+ return await replace(request, video_urls=[]).assemble_context()
supports_image = "image" in modalities
</code_context>
<issue_to_address>
**issue (broader_impact):** A fallback provider with an empty or missing `modalities` list receives a native `video_url` from the primary provider's already-assembled context. `sanitize_contexts_by_modalities` treats empty modalities as supporting every modality, while the runner's new opt-in rule only applies when initially assembling the request, so an unconfigured fallback can receive video and fail instead of receiving `[Video]`.
**Triggers:** When the primary provider opts into video but a fallback provider is unconfigured or has an empty `modalities` list.
**Suggested fix:** Apply the same video-specific opt-in rule in `sanitize_contexts_by_modalities`, filtering `video_url` blocks to `[Video]` when `video` is absent even if modalities is empty.
</issue_to_address>
### Comment 2
<location path="astrbot/core/provider/entities.py" line_range="228" />
<code_context>
+ content_blocks.append({"type": "text", "text": "[Video]"})
# 2. 额外的内容块（系统提醒、指令等）
if self.extra_user_content_parts:
</code_context>
<issue_to_address>
**issue (bug_risk):** A `VideoURLPart` supplied through `extra_user_content_parts` is appended without resolving it to base64 and without setting `_no_save`. The block is therefore persisted in conversation history, so a data-URI video remains a large history payload and a local or temporary video path is replayed after its source file is gone.
**Triggers:** When callers provide video through `extra_user_content_parts` rather than `ProviderRequest.video_urls`.
**Suggested fix:** Resolve `VideoURLPart` entries like `video_urls`, mark the resulting block transient, and ensure the provider-facing serialization still retains it.
</issue_to_address>Sourcery assessment
Needs a human reviewer. 2 findings to address first, and video attachments are now resolved, base64-encoded, and transmitted to providers that declare video support, so an incorrect eligibility or routing decision could expose user media or send it outside the team. Reverting would stop future transmissions but cannot retract video data already sent.
Blocking findings: astrbot/core/agent/runners/tool_loop_agent_runner.py:352, astrbot/core/provider/entities.py:228
CI note for reviewers: Root cause: SQLModel 0.0.45 (released 2026-09-21T21:51Z)
# sqlmodel/sql/sqltypes.py
def process_bind_param(self, value, dialect):
if value is None:
return None
if value.utcoffset() is None:
raise ValueError(
"Datetime values must have timezone information. "
"Use datetime.now(timezone.utc), or annotate the field with "
"NaiveDatetime for naive storage."
)Its EvidenceMaster is still at Bisect on this machine, running the failing tests with the venv interpreter (
Independently: every non- This PR's own tests are green
The latest run on Everything else on this PR is green: This job is not a merge gate
Fixing the underlying SQLModel regression is out of scope here (it needs |
Degrade `video_url` for an unconfigured provider on the fallback path. `sanitize_contexts_by_modalities` treats empty or missing `modalities` as "supports everything" for backward compatibility, but video is strictly opt-in. Since that function also sanitizes an already-assembled context when a request falls back to another provider, an unconfigured fallback could receive a `video_url` block that the primary provider produced, and fail the whole request instead of getting `[Video]`. Handle a `VideoURLPart` passed through `extra_user_content_parts` the same way as `video_urls`: resolve it to a portable data URI and flag it transient. Without that the block was persisted verbatim, so a data-URI video stayed in the conversation history as a multi-megabyte payload and a local path was replayed after its file was gone. A message left with nothing but transient blocks now keeps a text placeholder so it cannot persist as empty content. The unconfigured provider keeps receiving its original context objects when there is no video block to degrade, so the sanitizer is not entered and nothing else about that path changes.
…t-modality # Conflicts: # dashboard/src/assets/mdi-subset/materialdesignicons-subset.css # dashboard/src/assets/mdi-subset/materialdesignicons-webfont-subset.woff # dashboard/src/assets/mdi-subset/materialdesignicons-webfont-subset.woff2
Providers without native video input fell through to the catch-all �lse: raise ValueError(...) in their �xtra_user_content_parts loop, so a VideoURLPart raised instead of degrading. Video is opt-in and those APIs cannot accept it, so emit a [Video] text placeholder like every other unsupported modality and never persist the payload.
CI is green — master synced and the review findings are addressed
Head: CI on
|
| Check | Result |
|---|---|
Run pytest suite (ubuntu-latest) |
✅ pass — 3m23s |
Run pytest suite (macos-latest) |
✅ pass — 5m0s |
Run pytest suite (windows-latest) |
✅ pass — 6m51s |
format-check |
✅ pass |
build (Dashboard CI, incl. pnpm run i18n:check) |
✅ pass |
Analyze (python) / CodeQL |
✅ pass |
Smoke test — 15/15 (3 OS × Python 3.10–3.14) |
✅ pass |
windows-latest is continue-on-error in unit_tests.yml, so ubuntu/macos are the blocking lanes there.
What changed
1. Synced master. The earlier red Run pytest suite was not caused by this change: uv.lock is gitignored and sqlmodel>=0.0.24 has no upper bound, so uv sync --dev re-resolves SQLModel on every CI run, and SQLModel 0.0.45 started rejecting naive datetimes — breaking test_backup, test_dashboard, test_document_storage_fts, test_kb_upload_atomicity and test_stat_service (11 failures, reproducible on the base commit). master now carries 28c6aef1e fix: preserve datetime storage across SQLModel upgrades (#10211), which resolves all 11. The 11 upstream failures are gone; 0 failed.
2. Fixed a gap the merge surfaced (cc28030f4, 2 files, +16/−1). VideoURLPart is new in this PR, but only the Gemini path got an explicit branch for it. OpenAI and Anthropic fall through to the pre-existing catch-all else: raise ValueError(...) in their extra_user_content_parts loop, so a VideoURLPart raised instead of degrading — contrary to the opt-in/degrade contract this PR establishes. Both now emit a [Video] text placeholder, matching how they already handle AudioURLPart. Verified against the real provider classes:
| Case | Before | After |
|---|---|---|
OpenAI receives VideoURLPart |
❌ ValueError |
✅ [Video], no raise |
Anthropic receives VideoURLPart |
❌ ValueError |
✅ [Video], no raise |
| Genuinely unknown part type | raises | ✅ still raises (catch-all intact) |
ProviderOpenAIResponses inherits assemble_context from ProviderOpenAIOfficial (verified by identity), so one branch covers both OpenAI adapters.
3. Regenerated the MDI subset assets. The only merge conflict was in dashboard/src/assets/mdi-subset/ — icon-font subsets that subset-mdi-font.mjs rebuilds from src/ on every build, so both sides had produced different binaries. They were regenerated from the merged sources instead of hand-merged: the CSS is the union of both sides plus mdi-video-outline, and three independent regenerations are byte-identical. The new pnpm run i18n:check gate added on master passes too, with models.metadata.video present in all four locales.
Verification
tests/test_video_input_modality.py 23 passed
ruff format --check 516 files already formatted (uvx --from ruff==0.15.22)
ruff check All checks passed!
tests/ (full suite) 3844 passed, 68 skipped, 0 failed
dashboard: pnpm run i18n:check exit 0 (4131 leaf keys per locale, no drift)
dashboard: vue-tsc --noEmit exit 0
dashboard: pnpm run build exit 0
The local full-suite run deselects tests/test_local_shell_component.py and runs it separately: its child process gets an explicit stdin=PIPE (astrbot/core/computer/booters/local.py:376), which hangs under a non-TTY harness. 46 deselected == 31 passed + 15 skipped from that file, and 3844 + 68 + 46 = 3958 equals the collected total, so coverage is complete. That file's blob hash is identical across the PR head, the merge, and master — unrelated to this change.
The branch is a plain fast-forward over the previous head; nothing was force-pushed, and the pre-sync state is preserved at tag pr10202-presync-20261012.
Conflict with
|
…t-modality # Conflicts: # dashboard/src/assets/mdi-subset/materialdesignicons-webfont-subset.woff # dashboard/src/assets/mdi-subset/materialdesignicons-webfont-subset.woff2
Recurring conflict fixed at the root — the generated font subset is no longer part of this PRMerged latest This is the second time this PR conflicted, and both times the conflict was only in the auto-generated MDI icon subset — never in the feature. That is a sign the PR was fighting an upstream artifact rather than a real change, so I fixed the cause instead of resolving the binaries again. Why it keeps happening: What changed here: this PR no longer ships those three files at all — the merge takes Verified the build heals itself: running Verification on
Note: this merge also pulled in If you would rather keep these artifacts committed as a project convention, say so and I will restore them — but then this PR will conflict again on the next dashboard change. |
CI red is upstream, not this PR — and a fix is already open as #10617The three Root cause — reproduces on pristine
await shutdown_local_booter()
...
await asyncio.to_thread(close_trace_store)
if sp.db_helper is self.db:
await sp.close()The test mocks the managers and Verification that this is not caused by this PR:
Upstream fix already open: #10617 "fix: preserve trace snapshot ordering and stabilize cross-platform tests" — its description says "Fix the restart-test hang introduced after #10614", and it patches exactly this test to mock Everything else on this head is green: 20 of 23 checks pass, including
So this PR should go green once #10617 lands (or when master is otherwise fixed). No action from this PR is needed, and I did not add a second copy of that fix here to avoid colliding with #10617. |
This branch has not been deployed
1 participant
Adds video as a first-class input modality, mirroring the existing
image_urls/audio_urlsflow, so providers that declare it receive the video itself insteadof only a text placeholder.
Today
_append_video_attachment()only appends:so the model never sees any picture. On a real QQ deployment this pushed agentic
models into reading the file themselves (
astrbot_execute_python+cv2frameextraction,
ffprobe, web search for the dialogue), turning one video messageinto ~30 tool calls and ~12 minutes before a reply.
Requested in #8048. Supersedes #9424, which implements the same approach but is
conflicting with
masterand misses the two failure modes described below.Modifications / 改动点
Feature — video input modality
agent/message.py: addVideoURLPartand registervideo_urlin theContentParttype literal. Registration is required: the registry lookup inContentPart.__get_pydantic_core_schema__is unguarded, so an unregisteredblock raises
KeyErroras soon as it is validated from history.provider/entities.py: addProviderRequest.video_urls, count it in__repr__/_print_friendly_context, and emitvideo_urlblocks fromassemble_context().provider/sources/gemini_source.py: processvideo_urlblocks intoinline_dataparts, in both_prepare_conversationand the provider-sideassemble_context().astr_main_agent.py: collect the resolved video path intoreq.video_urlsalongside the existing text placeholder.
config/default.py: exposevideoas a model modality option.agent/runners/tool_loop_agent_runner.py,provider/modalities.py: includevideo in modality gating and degradation.
Fix 1 — the Gemini dispatch no longer has a catch-all
else_prepare_conversationpreviously ended with:A
video_urlblock fell into thatelseand raisedKeyError: 'audio_url'._prepare_conversationis awaited outside thetry:that guards the API call,so the
except APIErrornever saw it and the exception escapedtext_chat.Because the block is persisted into the session history, the failing message was
then replayed on every later request — one video would poison the whole session,
including plain-text turns. This is the same failure shape as #9295.
The branch is now exhaustive, and unknown block types degrade to a text
placeholder instead of raising.
Fix 2 — video blocks are marked transient
assemble_context()output is written intorun_context.messagesand replayedon every subsequent request, so a 20 MB video became ~27 MB of base64 resent each
turn. The video block now carries
_no_save, whichdump_messages_with_checkpointsfilters out. Providers still receive it, becausethey serialize messages with
model_dump(), which does not emit_no_save.Video is opt-in
An empty or missing
modalitieslist keeps its existing "supports everything"meaning for text/image/audio, but not for video. Video postdates that
convention and is not universally supported, so a provider that never opted in
would fail the entire request instead of degrading. Video is sent only when
videois explicitly present inmodalities.Fix 3 — video stays opt-in on the fallback path (review follow-up,
0274e3a)sanitize_contexts_by_modalitiestreats empty or missingmodalitiesas"supports everything", and
_sanitize_contexts_for_providershort-circuited onunconfigured providers. But that function also sanitizes an already-assembled
context when a request falls back to another provider — a context the primary
provider may have built with a
video_urlblock because it declared the videomodality. An unconfigured fallback therefore received the video and failed the
whole request instead of getting
[Video].Video is now degraded there too. The runner still hands back the original context
objects untouched when the provider is unconfigured and there is no video block
to strip, so nothing else about that path changes — entering the sanitizer
unconditionally converted every plain-text history to dicts, which broke six
existing
test_tool_loop_agent_runnertests.Fix 4 — a
VideoURLPartinextra_user_content_partsbehaves likevideo_urls(review follow-up,0274e3a)Such a part used to be appended verbatim: never resolved to a data URI, and never
marked transient, so a data-URI video stayed in the conversation history as a
multi-megabyte payload and a local path was replayed after its file was gone. It
now goes through the same
MediaResolver(..., media_type="video")call and isflagged
_no_save, and a message left with nothing but transient blocks keeps a[Video]text placeholder so it cannot persist as empty content.Dashboard
ProviderModelsPanel.vue,providerMetadata.ts,useProviderSources.ts,ProviderPage.vue,ProviderChatCompletionPanel.vue: add a video capabilitybadge. Unlike image/audio it is not reported as enabled for unconfigured
providers, matching the opt-in behaviour above.
i18n/locales/{zh-CN,en-US,ja-JP,ru-RU}: add themodels.metadata.videolabel.Regenerated MDI subset assets (the icon set changed).
This is NOT a breaking change. / 这不是一个破坏性变更。
Screenshots or Test Results / 运行截图或测试结果
Verification steps
uv sync --group devuv run pytest tests/test_video_input_modality.py -vuv run ruff format --check . && uv run ruff check .uv run pytest tests/ -qcd dashboard && pnpm install --frozen-lockfile && pnpm run buildResults
Update: master synced, CI is green
mastermoved forward while this PR was open, and the merge surfaced twothings worth calling out.
1. The upstream failure that made
Run pytest suitered is fixed upstream.The earlier red result was not caused by this change:
uv.lockis gitignoredand
sqlmodel>=0.0.24has no upper bound, souv sync --devre-resolvesSQLModel on every CI run, and SQLModel 0.0.45 (released 2026-09-21T21:51Z)
began rejecting naive datetimes — breaking
test_backup,test_dashboard,test_document_storage_fts,test_kb_upload_atomicityandtest_stat_service(11 failures, reproducible on that exact commit). Master now carries
28c6aef1e fix: preserve datetime storage across SQLModel upgrades (#10211),so syncing master resolves all 11. The full suite is green on all three
platforms (see the CI comment on this PR).
2. The merge exposed a gap in this feature, now fixed.
VideoURLPartis new in this PR, but only the Gemini path was given anexplicit branch for it. OpenAI and Anthropic fall through to the pre-existing
catch-all
else: raise ValueError(...)in theirextra_user_content_partsloop, so passing a
VideoURLPartraised instead of degrading — contrary to theopt-in/degrade contract this PR establishes. Those two providers now emit a
[Video]text placeholder, consistent with how they already handleAudioURLPart. The catch-all is untouched, andProviderOpenAIResponsesinherits the same method, so one branch covers both OpenAI adapters.
The only merge conflict was in the generated
dashboard/src/assets/mdi-subset/assets (icon font subsets rebuilt from
src/on every build). They wereregenerated from the merged sources rather than hand-merged — the resulting CSS
is the union of both sides plus
mdi-video-outline, and three independentregenerations produce byte-identical files. The new
pnpm run i18n:checkgateadded on master also passes, with
models.metadata.videopresent in all fourlocales.
The new tests cover, among others:
video_urldict validating through the registry withoutKeyErrorassemble_context()flagging the video block_no_savemodel_dump()(provider payload) while being absent fromdump_messages_with_checkpoints()(persisted history)_prepare_conversationaccepting avideo_urlblock and emitting aninline_datapart withmime_type == "video/mp4"replaced with
[Video]when not declaredthe rest of the history copied through untouched
VideoURLPartinextra_user_content_partsbeing resolved, markedtransient, kept out of history, and dropped when it cannot be resolved
Checklist / 检查清单
😊 If there are new features added in the PR, I have discussed it with the authors through issues/emails, etc.
/ 如果 PR 中有新加入的功能，已经通过 Issue / 邮件等方式和作者讨论过。
(Commented on [Feature] 增加视频输入支持 #8048 and feat: wire up video input modality for MiniMax providers #9424 with the design and the two failure modes.)
👀 My changes have been well-tested, and "Verification Steps" and "Screenshots" have been provided above.
/ 我的更改经过了良好的测试，并已在上方提供了“验证步骤”和“运行截图”。
📚 I checked the affected WebUI instructions and screenshots in
docs/zhanddocs/enagainst the changed navigation, page structure, and labels, and updated them in this PR (or explained why no documentation update is needed)./ 我已对照变化后的 WebUI 入口、页面结构和术语，核对并在本 PR 中更新
docs/zh和docs/en的相关操作说明与截图（或说明无需更新文档的原因）。(Checked: the model modality checkbox list is not documented in
docs/zhordocs/en— a search for模态/modalities/tool_usefinds only unrelated provider and config references. The navigation, page structure and entry points are unchanged, so no documentation update is required.)🤓 I have ensured that no new dependencies are introduced, OR if new dependencies are introduced, they have been added to the appropriate locations in
requirements.txtandpyproject.toml./ 我确保没有引入新依赖库，或者引入了新依赖库的同时将其添加到了
requirements.txt和pyproject.toml文件相应位置。(No new dependencies.)
😮 My changes do not introduce malicious code.
/ 我的更改没有引入恶意代码。
Summary by Sourcery
Support native video input as an explicitly enabled provider modality while safely degrading or excluding it for unsupported providers.
New Features:
Bug Fixes:
Enhancements:
Tests:
Chores: