Stage I: Visual Reasoning for Entity Alignment
1. Experimental Results on Accuracy of Entity Alignment
Left (Sec.4.1): Human reliance for the “Player Guessing” game. Human participants have an accuracy of 96.3%, among which 84.5% of the correct identifications can be made through various cues other than long view player tracking, indicating the necessity of involving fine-grained visual and contextual information in player alignment.
Right: (Appendix C.2) Ablation test on the contextual and visual components. Most components contributes to the visual reasoning task positively, while the KetEvent Timeline component is excluded in the following entity alignment experiments.
Left: (Exp.1) Zero-shot test on the video-LMMs and LLMs. LLMs fall shorts of video-LMMs in this visual reasoning task, indicating the importance of involving visual modality to further observe the related scene. Qwen2.5-VL serves as the superior backbone for further fine-tuning.
Right: (Exp.2) Fine-tuning on Qwen2.5-VL. The synergy between SFT and GRPO proves crucial for addressing the multifaceted challenges of visual reasoning in video-LMMs, improving the player alignment accuracy by 18.5% on SN-Caption-test-align dataset compared to Gemini 2.5-pro.
2. Case Study in Frame-Level Event Grounding Guidance (Appendix C.1)
Test set sample: [PLAYER] ([TEAM]) gets on the end of a pass on the edge of the box but his shot is blocked.
Train set sample: [PLAYER] ([TEAM]) goes over to take the corner kick and it looks like he will swing the ball into the penalty box.
3. Case Study in Visual Reasoning for Entity Alignment
| Video & Anonymized Commentary | Visual Reasoning for Entity Alignment |
|---|---|
|
[PLAYER] ([TEAM]) gets a shot away from around 24 metres out after his fine solo run. The ball went just wide of the right post!
(A good case by analyzing close view recognized face with player role.) |
Step 1: Video Grounding
The commentary event is described as occurring around 16:17 to 16:29 seconds in the video. The event involves a player from Barcelona taking a shot that goes wide of the goalpost. This suggests we need to focus on the Barcelona team's actions during this time frame. Step 2: Analyze the Attack and Defense Sides Given that the shot was taken by a Barcelona player, we need to identify who among their players fits the description of taking a shot. The Barcelona team has several attackers, including Lionel Messi (jersey number 10), Neymar (jersey number 11), and Luis Suarez (jersey number 9)✔️. Since the shot was taken from around 24 meters out, it is likely an attacking midfielder or forward who has possession and takes the shot. Step 3: Identify the Player Involved In the video, we see a player in a blue and red jersey (Barcelona) dribbling the ball towards the goal. This player is Neymar, as indicated by the close-up view showing his face and jersey number 11. The player's movement and positioning align with the description of taking a shot from a distance.✔️ Step 4: Context Knowledge Analysis From the history_timeline, we know that Lionel Messi was involved in an offside situation earlier. However, there is no direct evidence linking him to the shot described in the commentary. The close-up view of Neymar confirms his involvement in the event, as he is the only player from Barcelona seen in the close-up view during the relevant time frame.✔️ Final Deduction Based on the analysis, the player who took the shot and is mentioned in the commentary is Neymar from Barcelona. { "video_grounding": "16:17 - 16:29", "team": "Barcelona", "player_hash": "jNOPwl3h" , "player_name": "Neymar"} |
|
PLAYER] ([TEAM]) gets on the end of a pass on the edge of the box but his shot is blocked.
(A good case by observation in the long view and analyzing history timeline.) |
Step 1: Video Grounding
The commentary event is described as occurring around 15-17 seconds in the video. This is where the shot attempt is mentioned, and we need to focus on the scene around this time. Step 2: Analyze the Attack and Defense Sides The commentary mentions a "shot attempt" by a player from the home team, Paris SG. Given that Paris SG is playing against Arsenal, the attacking side would be Paris SG, and the defending side would be Arsenal. Therefore, the player who attempts the shot is more likely to be from Paris SG.✔️ Step 3: Identify the Player Involved In the video, around 15-17 seconds, there is a player in a blue jersey (Paris SG) who appears to be involved in an attacking play. This player is seen running towards the goal area with the ball. His jersey number is visible as 22, and his name is Jese,✔️ which matches the information given in the history_timeline. Step 4: Context Knowledge Analysis From the history_timeline, we know that Jese has been substituted into the game. This suggests he is actively participating in the match and is likely to be involved in significant plays, such as attempting shots on goal.✔️ Conclusion Based on the analysis, the player who conducted the shot attempt is Jese from Paris SG. His jersey number is 22, and his name is clearly identifiable in the video. {"video_grounding": "15-17", "team": "Paris SG", "player_hash": "6BdSXunk", "player_name": "Jese"} |
|
Poor challenge! [PLAYER] ([TEAM]) is penalised for tripping and [REFEREE] blows his whistle. [TEAM] win a free kick. It's a promising situation for a direct shot.
(A good case by analyzing close view recognized face with player role.) |
Step 1: Video Grounding
The commentary event is described as occurring around 14 19 21 27 29 seconds in the video. The closest grounded view to this time frame is around 22.52 seconds, where we see a close-up of a player wearing a white jersey. Step 2: Analyze the Attack and Defense Sides Given the context of the commentary, it mentions a foul that results in a free kick for the home team, Granada CF. This suggests that the player who committed the foul was likely from the opposing team, Real Madrid.✔️ The close-up view shows a player in a white jersey, which aligns with the away team's uniform. Step 3: Identify the Player Involved In the close-up view at 22.52 seconds, the player is wearing a white jersey. This player is identified as Pepe, who plays as a defender for Real Madrid. The close-up view confirms his identity, and he is the only player from Real Madrid visible in the close-up shots.✔️ Step 4: Context Knowledge Analysis The history_timeline indicates that a player from Real Madrid picked up a long through ball, but another player from the same team intercepted it. This suggests that players from Real Madrid were actively involved in the play leading up to the event. The close-up view of Pepe further supports his involvement in the event. Final Deduction Based on the close-up view and the context provided, the player who committed the foul is Pepe from Real Madrid. { "video_grounding": "22.52s", "team": "Real Madrid", "player_hash": "2y9qaipI", "player_name": "Pepe"} |
|
One of the defenders reacts at the last minute to brilliantly block a pass from [PLAYER] ([TEAM]).
(A bad case with wrong deduction.) |
Step 1: Video Grounding
The commentary event is described as occurring at around 14 15 16 29 seconds in the video. This suggests we need to focus on the scene where a defender reacts to block a pass from a player named [PLAYER]. Step 2: Analyze the Attack and Defense Sides Given that the event involves a defender blocking a pass, it's logical to look at the attacking side (Dortmund) and the defending side (Bayer Leverkusen). Since the defender is from Bayer Leverkusen, we need to identify who among their players fits the description of making a pass.✔️ Step 3: Identify the Player Involved In the video, we see a player in a red jersey (Bayer Leverkusen) passing the ball. The player's jersey number and name are visible, and he appears to be the one making the pass. The close-up view confirms that the player is indeed from Bayer Leverkusen, as indicated by the jersey color and design. Step 4: Context Knowledge Analysis From the history_timeline, we know that [PLAYER_O0EKKpNQ] received a yellow card for dissent.❌ The player's role as a midfielder suggests he could be involved in passing the ball. Final Deduction Based on the visual cues and the context provided, the player who made the pass and was blocked by a defender is identified as O0EKKpNQ from Bayer Leverkusen. { "video_grounding": "14 15 16 29", "team": "Bayer Leverkusen", "player_hash": "O0EKKpNQ" , "player_name": "Chicharito"} |
