LLLMWIKI
ArticleResearch mapReading portalMetadata

Source note · Updated 2026-09-25

REALMAn MLLM-Agent Framework for Open World 3D Reasoning Segmentation and Editing on Gaussian Splatting

Global-to-local MLLM grounding on prebuilt Gaussian scenes, with strong implicit-query scores and material preprocessing costs.

Read the paper

#near-cvpr-2025#3d-vision#gaussian-splatting#vision-language#primary-source

English translation pending — this page shows the Chinese original.

一句话结论

CVPR 2026在已有3DGS、训练相机和逐场实例特征场上，用Qwen2.5-VL+SAM跨全局与局部视角定位隐式语言所指物体；其高mIoU依赖大模型多次调用、视图选取与局部优化，8.68秒仅是单提示处理口径（主文pp.16782–16786 Tables1–4）。

输入、机制与评测

先由SAM训练视图掩码、跨视实例传播建立每高斯feature及分类器；全局阶段从24个相机簇选8个视角，经MLLM返回bbox、类别及理由、SAM出mask再投到ID场投票；局部阶段在目标可见视角做50步mask一致优化（主文Fig.3 Eq.1–8）。REALM3D有100场1444对提示掩码，Qwen2.5-VL生成隐式提示、SAM标注且人工过滤（主文Table2、补充pp.1–3）；同族MLLM对数据创建及解题有共因，需要独立人标场景级评测。

结果、反证与成本

  • 主文p.16784 Table1隐式提示LERF本法mIoU92.88 vs GS-Group42.43；3D-OVS 93.68 vs GAGS58.46；REALM3D 82.30 vs GS-Group65.55。§4.1只称LERF选2场/3D-OVS选5场重标，Table2原库列5/10场，主表评测分母需核。补充p.3 Table S1显示同3D-OVS显式提示GAGS84.18、本法93.62，对隐式能力说明任务更难；两者须同场景对照。
  • p.16786 Table3单一Figurines场：GS-Group mIoU.32，加Qwen2.5-VL .78，加全局 .89，加局部 .95；局部50步.95、500步.79，过拟合可见。Table4全局2.53秒+局部2.48秒+优化3.67秒；FPS354.72仅单通道掩码渲染，建图、实例场和LLM服务费另计。
  • 编辑Fig.7主要定性；补充pp.1–2：删除可接LaMa、替换接TRELLIS、风格化接InstructPix2Pix再优化1000步。成功率/保真度/墙钟及失败分母待独立核验。

可用性与相关页面

官方10页主文+6页补充、逐表审计存raw/ingest/2026-09-25-realm/。作者ChangyueShi/REALM仓库目前只有站点资源，未见可运行实现或LICENSE；未GPU复现。3D Gaussian Splatting: Representation, Optimization, Geometry, and Dynamic Scenes与LangSplat：高斯语言特征与三级语义查询并列语言任务，但逐场特征查询与多轮大模型推理的计算分母不同。

Metadata
{
  "id": "2026-09-25-realm",
  "type": "source",
  "title": "REALM：多视大模型推理定位与高斯实例编辑",
  "title_en": "REALM: An MLLM-Agent Framework for Open World 3D Reasoning Segmentation and Editing on Gaussian Splatting",
  "summary_en": "Global-to-local MLLM grounding on prebuilt Gaussian scenes, with strong implicit-query scores and material preprocessing costs.",
  "status": "reviewed",
  "note_status": "reviewed",
  "created": "2026-09-25",
  "updated": "2026-09-25",
  "source_type": "paper",
  "venue": "CVPR 2026",
  "authors": "[Changyue Shi, Minghao Chen, Yiping Mao, Chuxiao Yang, Xinyuan Hu, Jiajun Ding, Zhou Yu]",
  "published_at": "2026-06-01",
  "ingested_at": "2026-09-25",
  "canonical_links": [
    "https://openaccess.thecvf.com/content/CVPR2026/html/Shi_REALM_An_MLLM-Agent_Framework_for_Open_World_3D_Reasoning_Segmentation_CVPR_2026_paper.html",
    "https://openaccess.thecvf.com/content/CVPR2026/papers/Shi_REALM_An_MLLM-Agent_Framework_for_Open_World_3D_Reasoning_Segmentation_CVPR_2026_paper.pdf",
    "https://openaccess.thecvf.com/content/CVPR2026/supplemental/Shi_REALM_An_MLLM-Agent_CVPR_2026_supplemental.pdf",
    "https://github.com/ChangyueShi/REALM"
  ],
  "raw_entry": "raw/ingest/2026-09-25-realm/",
  "topics": "[topics/3d-gaussian-splatting]",
  "entities": [],
  "claims": [],
  "questions": [],
  "tags": [
    "near-cvpr-2025",
    "3d-vision",
    "gaussian-splatting",
    "vision-language",
    "primary-source"
  ]
}