*Equal contribution · †Corresponding author
BeyondCSe identifies objects or parts from past events, such as “the object moved last.” If the target is hidden, its earlier 3D positions guide the camera search. No task-specific training is required.
Each scene pairs one event history with several instructions. Select an instruction to see the robot grasp a different target. During execution, the robot uses only its wrist camera. The third-person recordings are provided for viewing, with execution shown at 2× speed.
A person moves a baseball, a red bell pepper, and a Rubik's cube in sequence, leaving them behind a box or inside a crate, out of the wrist camera's initial view. The instruction asks for the object moved first, second, or last. The robot identifies it from the video, then uses its recorded trajectory to select a view that reveals it before grasping.
The tools remain visible, but the instruction refers to an earlier action: driving a screw at the beginning or most recently. The robot reviews the video to determine which tool was used at that time and distinguishes it from similar-looking objects in the current scene.
The last object moved is a can, which ends up behind a blue crate. VLM Orchestrator (VoLo) identifies the can correctly, but π0.5 and Cosmos 3 fail to locate it in the current view. BeyondCSe moves the camera to see behind the crate, then grasps the can. The baselines use wrist and third-person cameras following the DROID setup; BeyondCSe uses only the wrist camera.
After observing a person moving or using objects, the robot receives an instruction that refers to those events. Such an event-referential instruction may be ambiguous from the current scene alone. BeyondCSe reviews the history to identify the requested object or part. If it is hidden, its earlier 3D positions provide an event prior: an initial estimate of where to search.

(A) VoLo identifies the can and passes its description to a VLA (π0.5) or a world action model (Cosmos 3). Both fail to grasp the hidden can in this example. (B) BeyondCSe uses the can's earlier 3D positions to estimate where it may be, then moves the camera to find and grasp it.
LERF-TOGO, GraspSplats, Point2Act, and GraspMolmo locate objects and parts from descriptions of the current scene. That scene alone cannot tell them which object was used or moved in an earlier event.
Memory VLAs and orchestrators such as VoLo can infer which object an instruction refers to. If it is now hidden, identifying it still leaves the robot with the task of finding a view from which it can grasp.
BeyondCSe recovers the target's past 3D trajectory and uses it to estimate where the object may be. This spatial belief guides camera movements and is updated as the robot observes the scene.
Video reasoning identifies the requested object or part and tries to locate it in the initial view. If it obtains a valid 3D target point, the system proceeds to grasp validation. Otherwise, it recovers earlier 3D observations and passes them to active perception, which selects new camera views to find the target. Expand either module for details.

A single pretrained multimodal language model (Qwen3-VL-8B) processes the video and instruction in four stages to select a pixel on the requested object or part. If no usable point is found in the initial view, a point tracker follows the target in earlier frames. Depth measurements convert these observations into a 3D event track 𝒫 that initializes the search.
Record the events in chronological order, without access to the instruction.
Use the instruction and event record to describe the target and the event that identifies it.
Use the cue to propose a bounding box in the initial image.
Select a target pixel within the crop and map it to 3D using depth and camera pose.

The region crop helps Point locate small parts, such as the flower stem shown here.

The event track and current scene geometry initialize a 3D probability distribution over possible target locations. Each camera view is scored by how much it could reduce uncertainty about those locations, accounting for the current belief. Views that depend on seeing through unobserved space are discounted, since that space may contain an occluder. New observations update both the scene map and target belief. Search ends when the target is found, no feasible view remains, or the view budget is reached.
Center a Gaussian at the last observed position and shape it using the end of the trajectory. A uniform component keeps other locations possible.
Rank feasible views by the target-belief mass they could downweight if the target is not seen, discounted by transmittance through unobserved space.
Update the belief using new depth measurements and a target-miss likelihood, reducing weights without ruling out locations entirely.
Check AnyGrasp candidates for target association and motion feasibility. If target geometry is incomplete, acquire a refinement view within the view budget.
We evaluate on a ROBOTIS OMY-F3M arm with a wrist-mounted RealSense D435i. All methods use the same robot, calibration, collision checks, and grasp pipeline.
Each method is evaluated on 50 visible-target and 100 occluded-target trials, except GraspMolmo (GM), which uses only the initial view and is evaluated only on visible targets. For the reasoned instruction condition, all baselines receive the same grounding queries generated by Qwen3-VL-8B from the history video. Each bar separates successful trials from failures at the localization, planning, or grasping stage; failures are counted at the first failed stage.

LERF*, P2A, GS, and GM denote LERF-TOGO, Point2Act, GraspSplats, and GraspMolmo, respectively. P2A† uses RGB-D reconstruction. Multi-view baselines receive 30 predefined views. BeyondCSe starts from the same initial view and selects additional views as needed.
Combined results for visible and occluded targets.
| Instruction | LERF* | P2A | GS | P2A† | Ours |
|---|---|---|---|---|---|
| Original | 0.0 | 9.3 | 0.0 | 22.7 | 76.7 |
| Reasoned | 18.7 | 19.3 | 34.7 | 50.0 |
Values are grasp success rates (%). BeyondCSe uses its own video reasoning in both instruction conditions.
This separate evaluation uses four additional heavily occluded scenes, with five trials per scene. The adapted method of Breyer et al. receives the ground-truth 3D target box; BeyondCSe uses the event history to identify and search for the target. S1–S4 and Views report mean view counts, including failed trials. Grasp reports success across all 20 trials.
| Method | S1 | S2 | S3 | S4 | Views ↓ | Grasp ↑ |
|---|---|---|---|---|---|---|
| Breyer et al. (GT box) | 4.2 | 4.2 | 2.2 | 2.8 | 3.35 | 15/20 (75%) |
| Ours w/o belief weights | 4.0 | 4.0 | 5.6 | 5.0 | 4.65 | 16/20 (80%) |
| Ours w/o transmittance | 2.8 | 2.0 | 5.2 | 2.4 | 3.10 | 18/20 (90%) |
| Ours w/o event prior | 3.8 | 4.0 | 4.2 | 4.0 | 4.00 | 19/20 (95%) |
| Ours (full) | 2.4 | 2.0 | 2.4 | 2.0 | 2.20 | 19/20 (95%) |
Replacing the event prior with a uniform initial belief increases the average view count from 2.20 to 4.00, while grasp success remains at 95% in these trials. In the qualitative example in the paper, BeyondCSe finds a view of the target after one move. The method of Breyer et al. takes several moves, even with the ground-truth target box.
Removing transmittance attenuation increases the average view count to 3.10, with one scene averaging 5.2 views. Removing belief weights gives all remaining location hypotheses equal weight and requires 4.65 views on average, with 80% grasp success.
Grounding queries generated from the video improve every baseline. The best baseline reaches 50.0% overall grasp success, compared with 76.7% for BeyondCSe. Localization remains the most common failure stage. BeyondCSe localizes 43/50 visible targets and 88/100 occluded targets, exceeding the strongest baselines by 32 and 20 percentage points, respectively.
The same event history that tells the robot which object to grasp can also help it find that object when it is hidden. BeyondCSe uses past observations to identify the target and guide the search, updating its spatial belief as it moves the camera. Across 150 real-robot trials, grasp success is 76.7%. A separate 20-trial evaluation on heavily occluded scenes achieves 95% grasp success with 2.20 views on average.
@article{lee2026beyondcse,
title = {Beyond the Current Scene: Event-Referential Grasping
with Active View Selection},
author = {Lee, Hyunjoon and Jung, Haebeom and Cha, Eunsung and
Lee, Daeun and Wang, Yu-Chiang Frank and Choe, Jaesung and
Park, Jaesik},
journal = {arXiv preprint arXiv:2609.39375},
year = {2026}
}