IEEE-CYBER 2024 · Finalist, Best Poster Award · Copenhagen
Dense books on a working deck leave no wrap-grasp envelope. The adapter locks a target with T-Match, queries an off-the-shelf grasp planner, and otherwise pushes a depth-filtered neighbor into the shelf until the target is graspable.
Front demo: one-take MuJoCo retrieval across four looks (library, office, warehouse, gallery), encoded at 4×. Perception PiP marks the working-row target and push point; the arm orbits 180° while pushing then lifting.
Retrieving a designated book from a crowded shelf is not a single-shot grasp. Neighbors close
the wrap envelope, so a planner that is competent in free space still returns no feasible
grasp. We implement the paper’s adapter as a closed loop over RGB-D: instance masks, a
persistent target identity (T-Match), a grasp planner Grsp(·) on the target cloud,
constrained approach evaluation, and a depth-filtered push that opens space beside the target
rather than cycling the same object.
The MuJoCo environment is a floor-standing bookcase. Only the bottom deck is interactive; upper decks are static fill. A dedicated eye camera and a 2D projection filter keep perception on that working row, which is what the front demo’s PiP shows.
One interaction is one pass around this loop. Grasp succeeds only after the working row has been opened; otherwise the push branch runs and the loop repeats.
Eye camera on the working deck. Depth + pinhole intrinsics for backprojection.
Detectron2 Mask R-CNN FPN (demo) or color heuristic. Keep masks that overlap interactive books.
Lock the red designated spine once; later steps MSE-match mask + appearance crop.
Target mask → camera-frame cloud → grasp candidates (AnyGrasp hook or heuristic).
Quality score plus geodesic proximity to shelf-axis default approaches.
If not graspable: nearest neighbor, depth filter, inward + lateral shove. Else wrap-grasp.
Package interactive_grasping (src layout). The adapter is backend-agnostic;
MuJoCo is the visual environment used on this page.
adapter.py
InteractiveObjectRetrievalAdapter.run_once is the closed loop. Observation
→ segment → optional working-row filter → T-Match → cloud → planner → constrained
score. A passing grasp calls execute_grasp; otherwise
DepthFilteredPushStrategy emits a PushAction.
target_match.pyFirst frame: nearest mean RGB to the designated red spine. Later frames: min MSE vs a buffer of target masks, plus weighted MSE of a 32×16 spine crop so identity survives after neighbors move.
push_strategy.pyNearest 2D centroid to the target, excluding the target. Depth filter drops objects that receded from the last push (already in the shelf). Contact point is the mask centroid backprojected to world; direction is inward (−Y) plus a lateral component away from the target.
grasp_evaluation.py
Sample default orientations about the shelf width axis (±45°, 80 samples). Score
1 − dgeo/π. Candidates below
constrained_score_threshold are treated as not yet graspable.
mujoco_env.pyUR5-style 6-DoF + parallel jaw, Jacobian IK. Floor-standing case; working deck at 0.162 m. Interactive books are free-jointed; deco books are welded. Eye camera hides the robot (geom group 3). Demo camera orbits 180° around the arm.
Detectron2FpnSegmentationBackend uses COCO Mask R-CNN R50-FPN, filtered to
class book, then NMS. Heuristic color blobs exist for tests. Grasp planner
default is heuristic; AnygraspBackend takes a planner_fn hook.
obs = env.get_observation()
parts = segment(obs.rgb) # Detectron2 or heuristic
parts = env.filter_interaction_masks(obs, parts)
k = tmatch.select_and_update(parts, rgb=obs.rgb)
cloud = mask_to_point_cloud(obs, parts[k].mask)
cands = planner.plan(cloud, parts[k].mask)
grasp = select_best_constrained(cands)
if grasp:
env.execute_grasp(grasp.pose)
else:
env.push(push_strategy.compute_push_action(...))
Suites live in scripts/run_experiments.py. Numbers below are from that script
on the MuJoCo bookcase with heuristic segmentation (so the table is reproducible without a
GPU). The hero video uses Detectron2. Grasp is admitted only after three opening pushes.
Loading experiment JSON…
Same adapter, four named looks, 12 interactive books, six seeds each.
Library look. Success should hold as the working row fills, at the cost of the required opening pushes.
no_push is the load-bearing ablation: without opening the row, wrap grasp is refused (0% here). no_tmatch and no_mask_filter still succeed under heuristic blobs because the simulator grasps the nearest interactive book. Those two knobs matter for Detectron2 PiP identity (hero demo), not for heuristic success rate.
Physics-free RGB rectangles, 20 seeds. Checks that the paper loop itself, not MuJoCo contact, is what needs the pushes.
Presented at the 14th IEEE International Conference on CYBER Technology in Automation, Control, and Intelligent Systems (IEEE-CYBER 2024), 16–19 July 2024, Copenhagen.
IEEE Xplore · doi:10.1109/CYBER63482.2024.10749016 · Author homepage
# front demo (Detectron2, writes demos/one_take.mp4)
.venv/bin/python scripts/run_one_take_demo.py --video demos/one_take.mp4 --seg detectron2
# 4× hero clip for this page
ffmpeg -i demos/one_take.mp4 -filter:v "setpts=0.25*PTS,fps=30" -an website/media/hero_4x.mp4
# tables on this page
.venv/bin/python scripts/run_experiments.py --out website/data/experiments.json --seeds 6
python -m http.server 8000 --directory website