ArticleMetadataMain page

topic · 2026-05-23

Video Understanding

A synthesis page for temporal reasoning, long-context video modeling, sports video understanding, and benchmark design.

#near-cvpr-2025#video-understanding#reasoning

概述

这一页追踪的不是单纯的视频分类，而是更复杂的时序表示、长上下文建模、视频-语言推理、体育视频中的目标关联，以及 benchmark 设计问题。当前证据已经覆盖通用 benchmark、失败模式诊断、长视频理解、编辑理解和一个新的体育 tracking 样本，因此这一页开始具备更清晰的问题轮廓：模型怎样处理长时序内容、拥挤场景下的轨迹维持、以及不同 benchmark 到底在测什么能力。

本页关注什么

  • 时序表示方法
  • 视频-语言理解
  • 长上下文视频推理
  • 体育视频中的目标跟踪与关联
  • benchmark 与评测范式变化
  • 视觉 backbone 与时序主干选型
  • 效率与时序保真之间的权衡

当前综合判断

topics/vision-backbones 现在提供基础架构入口：CNN / U-Net 负责局部感知和生成细节，Transformer / DiT 负责 token 化全局关系和生成 scaling，GCN / Skeleton Transformer 负责姿态动作结构，Mamba / SSM 负责长序列效率。

topics/sports-ai-video-understanding 已经把这页的体育 AI 分支抽成独立子线，topics/sports-ai-roadmap 进一步把这条子线整理成公开阅读路径与后续补强计划。sources/2026-04-24-sportsmot、sources/2026-04-24-teamtrack、sources/2026-04-24-sportshhi 和 sources/2026-04-24-transportmer 让这个分支从单个 SportMamba 样本扩展到 tracking benchmark、人-人交互、全场轨迹与多智能体推理。

sources/2026-04-23-sportmamba 把这页的证据边界向体育视频分析推进了一步。它说明视频理解层的问题并不只存在于 QA benchmark 或 video-language reasoning 里，拥挤场景下的 player tracking 同样要求模型处理复杂时序依赖、遮挡恢复和非线性运动。

entities/sportsmot 的加入也让这一页第一次有了更具体的体育视频数据锚点：团队运动场景里的时序理解常常要落到“能否持续跟住同一个人”这种结构化任务上，而不只是回答问题或打分。

sources/2026-04-14-sti-bench 进一步把“精确时空世界理解”拉成单独 benchmark 维度，使这一页对精细时空理解的覆盖更完整。

sources/2026-04-14-lvbench、sources/2026-04-14-sok-bench 和 sources/2026-04-14-anetqa 让这一页终于从“有一些样本”推进到“有更完整 benchmark 结构”的阶段：长视频、情境推理、未裁剪组合推理都开始有对应支点。

sources/2026-04-14-mvbench 让这一页第一次有了更像“主基线”的综合 benchmark，而不只是 failure diagnosis 样本。

sources/2026-04-14-language-driven-video-inpainting 也说明视频理解已经开始更直接地嵌入编辑型任务接口，而不是只留在纯 benchmark 页面里。

sources/2026-04-14-glitchbench 又补入了动态异常识别这一类诊断 benchmark，使这一页不再只停留在概念层。

sources/2026-04-14-veggie 说明视频理解正在和 instruction-based editing 直接耦合；sources/2026-04-14-hallusionbench 则把多模态理解失败模式的诊断层补了进来。

这一主题现在已经可以承载更明确的系统观察：核心问题是模型是否真正理解长时序内容、动作关系、事件演化，以及在真实拥挤场景里能否持续保持稳定的对象级理解与关联。体育机器视觉文献清单 进一步把体育视频理解和部署协议、跨场景泛化、可解释输出、人工复核连接起来；sources/2026-05-05-soccernet-v2-holistic-understanding、sources/2026-05-05-soccernet-v2-camera-calibration-player-localization、sources/2026-05-05-active-learning-action-spotting-football、sources/2026-05-05-soccernet-2022-challenges-results、sources/2026-05-05-soccernet-2023-challenges-results 与 sources/2026-05-05-soccernet-2023-tracking-mot4mot 把足球视频理解补到数据生态、球场几何、低标注事件定位、年度挑战和 tracking 系统；sources/2026-05-05-human-in-loop-team-sports-retrieval、sources/2026-05-05-finecausal-action-quality-assessment 与 sources/2026-05-05-ai-driven-soccer-analysis 则把这条线继续推进到人机闭环、因果解释和应用系统；sources/2026-05-16-bst-badminton-stroke-type-transformer、sources/2026-05-12-trackmae 与 sources/2026-05-12-videoauto-r1 进一步把通用视频理解接到体育动作语义和动作纠正：BST 负责羽毛球击球类型，TrackMAE 负责运动轨迹表征，VideoAuto-R1 负责按需解释。

证据基础

体育多智能体轨迹：事件语义作为长视频上下文

sources/2026-05-05-event2tracking 给 video understanding 增加了一个强应用锚点：长视频理解可以服务于多智能体轨迹重建，event data 能成为恢复 noisy trajectories 的上下文。这与 topics/sports-ai-video-understanding 中的 SoccerNet challenge、tracking team report 和 game-state reconstruction 证据形成连续链路。

体育裁判推理：专业规则作为视频理解压力测试

sources/2026-05-05-x-vars-explainable-football-refereeing、sources/2026-05-05-refereebench-multi-sport-referees 与 sources/2026-05-05-sports-qa-video-question-answering 给 video understanding 增加了一个专业评测方向：模型需要把视频片段、实体、时间点、运动规则和解释链合在一起。这个方向比通用 VideoQA 更贴近真实决策，也更能暴露 MLLM 在 temporal grounding、规则遵循和可解释输出上的短板。

高速小目标追踪：体育视频里的极端 temporal localization

sources/2026-05-05-tracknet-high-speed-tiny-objects、sources/2026-05-05-tracknetv2-efficient-shuttlecock-tracking、sources/2026-04-25-tracknetv3、sources/2026-05-05-tracknetv4-motion-attention-maps 与 sources/2026-05-05-monotrack-shuttle-trajectory-reconstruction 给 video understanding 增加了一个极端应用：目标很小、速度很快、模糊和遮挡频繁，但轨迹又直接决定战术和训练分析价值。

细粒度体育动作语义：从骨架动作识别到击球类型

sources/2026-05-16-bst-badminton-stroke-type-transformer 给 video understanding 增加了一个细粒度动作语义样本：系统需要把人体骨架、shuttle trajectory 和场上位置融合起来，预测羽毛球 stroke type。sources/2026-05-16-shuttleset-stroke-level-badminton-dataset 补上 stroke-level 数据结构，sources/2026-05-16-tempose-badminton-fine-grained-motion 补上 badminton skeleton Transformer 方法锚点，sources/2026-05-16-blockgcn-topology-aware-skeleton-action-recognition 补上 topology-aware GCN，sources/2026-05-16-skateformer-skeletal-temporal-transformer 补上 skeletal-temporal Transformer，sources/2026-05-16-protogcn-skeleton-action-recognition 则说明通用 SAR 正在把相似动作的局部细节差异建模成 prototype-level representation。

这条线和 sources/2026-04-25-st-gcn 形成方法前史关系，也和 sources/2026-05-12-trackmae 的 motion-aware representation 形成后续动作纠正接口。

动作纠正：运动轨迹表征与按需解释

sources/2026-05-12-trackmae 和 sources/2026-05-12-videoauto-r1 给 video understanding 增加了一个更接近产品的评估维度：模型是否能把动作细节、错误阶段和解释策略连接起来。TrackMAE 指向 motion-centric representation，VideoAuto-R1 指向 reason-when-necessary policy；它们共同说明体育动作纠正的核心不只是分类精度，还包括可解释性、置信度和反馈成本。

关联页面

后续值得追踪的问题

  • 当前 benchmark 是否奖励了大量浅层时序捷径？
  • 模型在长视频里如何避免丢失关键时序细节？
  • 体育视频中的 tracking 表现能否转化成更强的高层事件理解能力？
  • 体育 AI 是否应优先从轨迹/交互/战术状态这些结构化中间层切入？
  • 视频语言方法相比纯视频方法，真正的优势边界在哪里？
  • 体育动作纠正中，哪些错误只需要感知判断，哪些错误必须进入因果/推理解释？
  • 羽毛球这类高速小目标项目中，stroke-type classification 能否稳定承接 TrackNet / MonoTrack 的球路输出？这条问题已推进到 questions/question-badminton-stroke-correction-demo：用 stroke type、关键帧、错误模式和反馈模板定义最小可复现 demo。

Metadata

{
  "id": "topic-video-understanding",
  "type": "topic",
  "title": "视频理解",
  "title_en": "Video Understanding",
  "status": "active",
  "created": "2026-04-12",
  "updated": "2026-05-23",
  "tags": [
    "near-cvpr-2025",
    "video-understanding",
    "reasoning"
  ],
  "summary": "追踪视频系统中的时序推理、长上下文建模、体育视频跟踪、体育 AI 视频理解与 benchmark 设计问题。",
  "summary_en": "A synthesis page for temporal reasoning, long-context video modeling, sports video understanding, and benchmark design.",
  "source_notes": [
    "sources/2026-04-14-veggie",
    "sources/2026-04-14-hallusionbench",
    "sources/2026-04-14-glitchbench",
    "sources/2026-04-14-language-driven-video-inpainting",
    "sources/2026-04-14-mvbench",
    "sources/2026-04-14-lvbench",
    "sources/2026-04-14-sok-bench",
    "sources/2026-04-14-anetqa",
    "sources/2026-04-14-sti-bench",
    "sources/2026-04-23-sportmamba",
    "sources/2026-04-24-sportsmot",
    "sources/2026-04-24-sportshhi",
    "sources/2026-04-24-teamtrack",
    "sources/2026-04-24-transportmer",
    "sources/2026-04-25-finegym",
    "sources/2026-04-25-multisports",
    "sources/2026-04-25-finediving",
    "sources/2026-04-25-logo-group-action-quality",
    "sources/2026-04-25-tacticai",
    "sources/2026-04-25-soccernet-game-state-reconstruction",
    "sources/2026-04-25-finesports",
    "sources/2026-04-25-athletepose3d",
    "sources/2026-04-25-universal-soccer-video-understanding",
    "sources/2026-04-25-tracknetv3",
    "sources/2026-04-25-wear-outdoor-sports-dataset",
    "sources/2026-04-25-soccernet-action-spotting",
    "sources/2026-04-25-sportscap",
    "sources/2026-04-25-automated-offside-detection",
    "sources/2026-04-25-opencap",
    "sources/2026-04-25-temporal-parsing-transformer-aqa",
    "sources/2026-04-25-timesformer",
    "sources/2026-04-25-st-gcn",
    "sources/2026-04-25-videomae",
    "sources/2026-04-25-sportsmot-dataset-page",
    "sources/2026-04-25-finediving-repository",
    "sources/2026-04-25-athleticspose",
    "sources/2026-04-25-sports-ai-review-2025",
    "sources/2026-04-25-ap-nfl-hawkeye-first-down",
    "sources/2026-04-25-mmaction2",
    "sources/2026-04-25-reuters-premier-league-semi-automated-offside",
    "sources/2026-04-25-genius-sports-second-spectrum",
    "sources/2026-04-25-catapult-athlete-monitoring",
    "sources/2026-04-25-kinexon-player-tracking",
    "sources/2026-04-25-homecourt",
    "sources/2026-04-25-openpose",
    "sources/2026-04-25-mmpose",
    "sources/2026-04-25-mediapipe-pose",
    "sources/2026-04-25-deeplabcut",
    "sources/2026-04-25-nist-ai-risk-management-framework",
    "sources/2026-05-05-human-in-loop-team-sports-retrieval",
    "sources/2026-05-05-finecausal-action-quality-assessment",
    "sources/2026-05-05-ai-driven-soccer-analysis",
    "sources/2026-05-05-soccernet-v2-holistic-understanding",
    "sources/2026-05-05-soccernet-v2-camera-calibration-player-localization",
    "sources/2026-05-05-active-learning-action-spotting-football",
    "sources/2026-05-05-soccernet-2023-challenges-results",
    "sources/2026-05-05-soccernet-2022-challenges-results",
    "sources/2026-05-05-soccernet-2023-tracking-mot4mot",
    "sources/2026-05-05-event2tracking",
    "sources/2026-05-05-x-vars-explainable-football-refereeing",
    "sources/2026-05-05-refereebench-multi-sport-referees",
    "sources/2026-05-05-sports-qa-video-question-answering",
    "sources/2026-05-05-tracknet-high-speed-tiny-objects",
    "sources/2026-05-05-tracknetv2-efficient-shuttlecock-tracking",
    "sources/2026-05-05-monotrack-shuttle-trajectory-reconstruction",
    "sources/2026-05-05-tracknetv4-motion-attention-maps",
    "sources/2026-05-12-trackmae",
    "sources/2026-05-12-videoauto-r1",
    "sources/2026-05-16-bst-badminton-stroke-type-transformer",
    "sources/2026-05-16-tempose-badminton-fine-grained-motion",
    "sources/2026-05-16-shuttleset-stroke-level-badminton-dataset",
    "sources/2026-05-16-protogcn-skeleton-action-recognition",
    "sources/2026-05-16-blockgcn-topology-aware-skeleton-action-recognition",
    "sources/2026-05-16-skateformer-skeletal-temporal-transformer"
  ],
  "related_topics": [
    "topics/vision-language",
    "topics/generative-model-evaluation",
    "topics/diffusion-models",
    "topics/vision-backbones",
    "topics/sports-ai-video-understanding",
    "topics/sports-ai-roadmap"
  ],
  "related_entities": [
    "entities/sportsmot"
  ],
  "open_questions": [
    "questions/question-badminton-stroke-correction-demo"
  ]
}