Although GPT series models perform relatively better, they exhibit notable deficiencies in certain areas, highlighting the need for further improvements and optimizations. Our assessment of popular API-based and open-source MLLMs reveals that these models natural language processing still face significant challenges in processing egocentric videos. This demonstrates that generating feedback requires more fine-grained visual information not present in captions and superior reasoning capability.
At the core of Hier-EgoPack is Temporal Distance Gated Convolution (TDGC), a novel GNN layer for egocentric vision tasks that require a strong sense of time, i.e., the ability Pegasus 1.6 Targets the Data Work Behind Robot Training to effectively reason on the order of the events in a video. In the second stage, the model is fine-tuned on a novel task using Hier-EgoPack’s cross-task interaction mechanism. Requests for name changes in the electronic proceedings will be accepted with no questions asked.
API-based and image-based MLLMs abandon this information after extracting keyframes, necessitating the use of open-source video-based MLLMs for performance assessment. We comprehensively select the most popular and high-performance video-based MLLMs to evaluate. large language models Below, we provide a brief introduction and present their detailed component information in Table 3. In this section, we mainly introduce our extensive adopted models, including API-based models, a series of open-source image-based and video-based MLLMs.
Step 2: Novel Task Learning
- Therefore, we mainly adopt open-ended text generation as our task format for traditional video question-answering.
- Ego-ExoLearn (Huang et al., 2024) and Ego-Exo4D (Grauman et al., 2024) emphasize egocentric videos that demonstrate an individual’s understanding of activities when given an exocentric demonstration.
- As more egocentric videos become available, specialized datasets focusing on specific aspects of ego-perspective have emerged.
- EgoThink (Cheng et al., 2024) covers more comprehensive capabilities but uses static images.
- Two mainstream methods for video question-answering include multiple-choice and open-ended question-answering.
- However, recent advancements in API-based and video-based MLLMs (Achiam et al., 2023; Anthropic, 2024; Reid et al., 2024; Li et al., 2023c; Lin et al., 2023) have demonstrated significant progress.
- However, these metrics are a trade-off due to the challenges of evaluation video planning tasks.
For object grounding and temporal grounding, we use mean intersection over union (mIoU) as the uniform metric, named mIoU-Object and mIoU-Temporal, respectively. However, these metrics are a trade-off due to the challenges of evaluation video planning tasks. Therefore, we mainly adopt open-ended text generation as our task format for traditional video question-answering. Two mainstream methods for video question-answering include multiple-choice and open-ended question-answering. However, recent advancements in API-based and video-based MLLMs (Achiam et al., 2023; Anthropic, 2024; Reid et al., 2024; Li et al., 2023c; Lin et al., 2023) have demonstrated significant progress.
To underscore the differences between static images and dynamic videos (Li et al., 2023d), we emphasize temporal attributes, ensuring that questions require the entire video for accurate answers rather than just a single frame. These datasets provide a robust foundation for training and evaluating MLLMs from a first-person perspective. For instance, LEMMA (Jia et al., 2020) includes data on goal-directed actions and multi-task situations. Egocentric video datasets (Grauman et al., 2022; Damen et al., 2018; Pirsiavash & Ramanan, 2012; Sigurdsson et al., 2018) capture first-person interactions with environment, aiding robotic tasks and augmented reality. A comparison to recent video-language benchmarks is presented in Table 1.
- In this work, we explore egocentric video understanding capabilities of MLLMs.
- Moreover, all these egocentric benchmarks with only conventional VQA tasks neglect that the designed task format should be grounded in the potential applications.
- While MLLMs excel at visual perception and reasoning in third-person and egocentric videos, they are prone to hallucinations, generating coherent yet inaccurate responses.
- By doing so, you can easily group experiment by name on the WandB dashboard and evaluate the metrics on the average of three different runs, to obtain more consistent results.
- The bold font denotes the best performance and the underline font denotes the second-best performance.
Incentivizing Temporal-Awareness in Egocentric Video Understanding Models
For example, multi-task learning can share weights across different tasks, but it does not explicitly model cross-task sinergies and can lead to negative transfer between tasks. Multi-task learning shares a common backbone among the different tasks, with small task-specific heads on top. In addition, used as a tool of re-memory cognitive training, lifelogs would enable the prevention of cognitive and functional decline in elderly people. In both cases, since the camera is worn in a naturalistic setting, visual data present a huge variability in terms of illumination conditions and object appearance.
- Consequently, visual grounding has garnered significant attention in both image- and video-based MLLMs.
- The bold font denotes the best performance and the underline font denotes the second-best performance.
- In this work, we explore egocentric video understanding capabilities of MLLMs.
- From the wide selection of open-source LLMs, numerous image-based MLLMs (Chen et al., 2023b; Liu et al., 2024b; Zhang et al., 2023; Dai et al., 2024; Alayrac et al., 2022) have emerged.
- However, recent advancements in API-based and video-based MLLMs have demonstrated significant progress.
Submitted on 9 Sep 2025 (v1), last revised 14 Oct 2025 (this version, v2) Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. Our benchmark will be open-sourced for https://aicurated.com/ reproducibility While MLLMs excel at visual perception and reasoning in third-person and egocentric videos, they are prone to hallucinations, generating coherent yet inaccurate responses. Activations consensus between two support tasks is defined as the percentage of their prototypes corresponding to the same label activated by the two tasks. LTA and OSCC have more uniform activations across different support tasks, i.e., they look at similar prototypes, while MQ exhibit more diverse activations.
MM-Ego: Towards Building Egocentric Multimodal LLMs – Apple Machine Learning Research
MM-Ego: Towards Building Egocentric Multimodal LLMs.
Posted: Thu, 10 Apr 2025 22:33:16 GMT source
Each negative sample includes three feedback demonstrating the incomplete action. Our clipped videos in the hierarchy planning task contain entire mid-level steps, which we use to construct the reward modeling dataset. Due to the noise in generated instances and the cost of API-based evaluation, three human annotators filter them to ensure quality and diversity, selecting the most representative examples. We develop specific prompts, combined with the narrations, as inputs for GPT-4o tailored to each fine-grained dimension. Consequently, visual grounding (Peng et al., 2023; Chen et al., 2023a; Munasinghe et al., 2023) has garnered significant attention in both image- and video-based MLLMs. We will introduce an advanced version in future work, as discussed in Sec. 6.

