OCT. 13, 2026Kanishka Rao Principal Software EngineerGemini 4 Argon marks a step change in physical understanding, code generation, and visual-semantic reasoning—unlocking new workflows across the robotics stack and accelerating progress toward physical AGI.You can see this leap in capability on the Embodied Reasoning (ERQA) benchmark, which tests a model's ability to understand and solve physical, spatial problems in 3D environments. Using pure reasoning, Argon scores 87%, outperforming every other frontier model.As a general-purpose physical reasoning model, Argon can be applied across the entire robotics lifecycle. In this post, we share representative prompts for Argon across four key areas:Robot control → Mobile manipulation with MuJoCo practice, zero-shot robot control, and closed-loop steering of video world models.Real2Sim → Reconstructing physically calibrated simulation twins from robot camera feeds, egocentric human video, or hand-drawn paper sketches.Physical understanding → Agentic multimodal hindsight labeling including detailed understanding of dexterity, dense value functions and progress estimation.Research and self-improvement → Action improvement via simulation and iterative self-improvement on hardware.Unlocking robot control with Argon1. Spot mobile manipulation: Putting in the trash bin with MuJoCo practiceArgon controls a Boston Dynamics Spot® quadruped (12-DoF legs + 6-DoF arm) to learn a new skill: stepping on a foot-pedal trash bin to throw away soda cans. Argon creates an articulated MuJoCo digital twin of a foot-pedal trash bin to practice its moves before stepping on the physical pedal and dropping soda cans into the open bin.Sorry, your browser doesn't support playback for this videoSpot Mobile Manipulation — Real-to-Sim-to-Real foot-pedal trash bin disposal: Spot picking up the soda can, checking its wrist camera, practicing 3-leg tripod balance in MuJoCo, stepping on the foot pedal, verifying hand centering in the open bin, and dropping the can inside.Prompt:There's a Diet Coke can behind you and another on the black table on your left.Use the spot-mobile-manipulation skill to pick them up and trash them in the black foot-pedal bin.Before stepping down on the pedal, verify in the front cameras that your front-right foot target is centered on the pedal, and use a 2-agent verification swarm (foot-verifier and hand-verifier) to verify both the foot-pedal step and the 3D hand-in-bin alignment before releasing.Plain textCopied 2. Zero-shot controlArgon controls a simple robotic arm to place a fallen mug upright and put the crumpled napkin inside. Argon reasons with a visual coordinate harness and action primitives to perform this task over several turns.Sorry, your browser doesn't support playback for this videoLeft: 3rd-person real robot view; Right: calibrated wrist camera with projected 2 cm world coordinates and jaw-closing reticle.Prompt:[Attached: wrist_cam.png, obs.json]Task: Get the robot arm to reorient the mug upright and put the napkin inside the mug.## Execution Discipline- Simply output 1 sentence observation followed immediately by your `curl` command (with `# Plan:` on Turn 1 and `# Thought:` on every turn). Everything needed is in `obs.json` and the calibrated reticle/grid drawn on `wrist_cam.png`.## Visual Coordinate Reference (Annotated on wrist_cam.png)- `+X` (UP in image): Away from robot base | `-X` (DOWN): Towards robot base- `+Y` (LEFT in image): World left | `-Y` (RIGHT): World right- `+Z`: Height above table | `Yellow numbers`: World (X, Y) in centimetres (2 cm grid at `grid_z`)- `Cyan + crosshair`: Current 3D pinch center- `Cyan [ + ] brackets`: Current jaw opening and closing line- `Green line`: Current object long-axis orientation (`yaw_deg`: 0° = vertical along +X, 90° = horizontal along +Y)- `pitch_deg = 90°`: Straight down (-Z) top-down grasp/place; `pitch_deg < 60°`: Tilted coaxial approach along finger axis## Closed-Loop Turn Workflow1. Inspect current state & view `wrist_cam.png`: `curl -s http://127.0.0.1:8081/obs` (returns `ee_pos`, `holding`, `status`, `workspace`, `wrist_cam_path`).2. Select VLM call granularity upfront on Turn 1: - Coarse (2–4 calls): `look_at` -> `grasp` -> `place` -> `done` - Pre-Grasp Hover Verify (4–7 calls): `look_at` -> `goto(z=0.04)` (verify cyan `[ + ]` brackets straddle target) -> `grasp` -> `place_until_contact` -> `done` - Fine Visual Servoing (8–20 calls): `look_at` -> `grasp` -> pre-pose `goto` -> closed-loop `move_by` / `move_until_contact` -> `done`3. Send motion primitives (`POST /plan`) and verify visually after every turn in the updated `wrist_cam.png`. If `status == "grasp_failed"`, call `robot.look_at(...)` to re-observe and recover closed-loop.## Code-as-Policies Primitives (`POST http://127.0.0.1:8081/plan`)robot.look_at(x=0.35, y=0.13, z=0.14, grid_z=0.015)x, y = robot.deproject(u=0.52, v=0.48, z_plane=0.015)robot.grasp(x=x, y=y, z=0.015, yaw_deg=-35, pitch_deg=90)robot.place(x=0.24, y=0.06, z=0.075)robot.place_until_contact(x=0.24, y=0.06, z=0.015, max_lag=0.012, relief=0.002)robot.goto(x=0.28, y=0.05, z=0.04, arc_z=0.14, pitch_deg=25, azimuth_deg=90, roll_deg=0, speed=0.10)robot.move_by(forward=0.025, dz=-0.002, dpitch_deg=-3, droll_deg=5, speed=0.04)robot.move_until_contact(dz=-0.06, speed=0.03, max_lag=0.012, relief=0.002)robot.open_gripper() / robot.close_gripper() / robot.home() / robot.done()Plain textCopied 3. Omni controlArgon steers robots in the Gemini Omni World Model to rearrange five scattered Scrabble tiles (G, R, A, N, O) into a clean row spelling A - R - G - O - N. Instead of specifying one static prompt, Argon steers in a closed visual loop. It writes timestamped action prompts, inspecting intermediate rollout keyframes, and refining its prompts.Sorry, your browser doesn't support playback for this videoIterative Video World Model Steering Dexterous Hands: 33-Step Steered Video Rollout Spelling A-R-G-O-N.Prompt:Goal: {task_goal}You are steering a video world model (Gemini Omni) in an iterative visual feedback loop. Inspect the attached frames from the latest rollout and write the next timestamped action prompt for the video world model:- Break complex motions into one simple, single-direction action at a time.- Keep inactive hands retracted and avoid blocking the camera's view of objects in the workspace.- Explicitly state which areas of the workspace become empty after an object moves.- Use [MM:SS.mmm] timestamps for each action beat and wrap your output in tags.Example format (sample timestamps only, not the full prompt):[00:00.000] Character A's left hand rests at the side while the right hand reaches toward the target object.[00:02.500] Character A's right fingertips push the object smoothly into the center of the table, leaving its starting area empty.[00:05.000] Character A's right hand opens and retracts to the side so all placed objects remain in clear view of the camera.Plain textCopied Unlocking Real2Sim with Argon1. Robot view to simulationArgon views a real robot teleoperation episode and develops a digital twin; it obtains or creates simulation assets that look and behave like the real objects upon contact with the dexterous hands. Argon tunes the assets and scenes to match sim visuals and contact events with those in the real recorded data in an iterative visual-reasoning loop.Sorry, your browser doesn't support playback for this videoLeft (Real View): synchronized real robot camera feeds; Right (Simulated): 3D MuJoCo scene reconstructed and calibrated by Argon.Prompt:[Attached: robot_rollout_episode]Build a MuJoCo scene of the robot and reconstructed task objects. When each episode is replayed frame by frame, by driving the robot joints toward the logged observation.state with joint-space PD.Iterate using two verification objectives:(1) Visual match: MuJoCo renders from the head_left, left_wrist, and right_wrist cameras line up with, and look like, those real observations.(2) Physical match: the simulated contact force/torque on each Sharpa fingertip, in the sensor frame, matches the logged tactile_force in timing, magnitude and direction.What you can tune happens before simulation starts:(1) Static object assets: realistic geometry, collision shapes, mass, inertia, friction and joints, with appearance close to the real objects, plus the initial object poses at t = 0.(2) Offline-planned robot commands, changed only in frames where the real fingertips are near an object. Each command must stay within 0.55 rad (arm joints) and 0.65 rad (hand joints) of the logged teleop trajectory. Everywhere else, replay the logged trajectory unchanged.No sim cheating: task objects may move only through real MuJoCo contact with the robot hands and the environment (mj_step).Not allowed after t = 0: changing MjModel parameters; applying external forces or torques to objects; welds or equality constraints; feedforward torques beyond joint-space PD; setting or rolling back object qpos/qvel; disabling or excluding collisions; reporting contact forces that don't come from the MuJoCo solver (for example, copying the logged tactile readings).Immediately present the reconstructed scene and rollout results as an interactive HTML session viewer with one page per episode, showing:- At the bottom: A synced timeline (play, scrub, speed control) that drives every panel.- On the left panel: An interactive 3D Blender scene showing the simulated robot and objects, showing contact points and force arrows as replay progresses.- On the right panel: Real vs. sim camera views side by side for all 3 cameras. Real vs. sim finger force visualizations.Plain textCopied 2. Human view to simulationGiven an egocentric human video of a manipulation task, Argon identifies the task-relevant objects, replaces crude bounding boxes with compound convex-decomposed meshes for concave containers, and retargets the human hand trajectory onto a 76-DoF bimanual humanoid with 22-DoF five-fingered hands in MuJoCo contact physics.Sorry, your browser doesn't support playback for this videoHuman View to Simulation — Egocentric Human Video to Dexterous Humanoid Twin: Left shows the input egocentric human RGB video; Right shows the synchronized 500 Hz MuJoCo + Blender Cycles multi-view simulation rollout.Prompt:[Attached: egocentric_human_video.mp4, candidate_sim_views.png, hand_3d_keypoints.json]You are building a physically solvable 500 Hz MuJoCo manipulation environment (`scene.xml`) and studio Blender render that faithfully reproduces the tabletop task shown in `egocentric_human_video.mp4` on a bimanual humanoid robot with 22-DoF dexterous five-fingered hands:1. Scene Reconstruction & Camera Calibration: - Identify only the manipulation-relevant objects (the grasped target object and the receiving container or landmark) and triangulate their 3D tabletop coordinates from the egocentric camera intrinsics/extrinsics. - Replace crude bounding boxes with compound convex-decomposed 3D meshes so concave containers (nested bowls, recessed wooden trays, open bins) support physical interior placement.2. 500 Hz Dexterous Motion-Fitting Stress Test: - Retarget the 3D human wrist and finger keypoints to the humanoid's 7-DoF arm and 22-DoF hand joints, and step the trajectory in MuJoCo at 500 Hz (dt = 0.002s) with zero kinematic weld constraints. - Verify opposing thumb-and-finger force closure, clean object lift (> 2 cm), and stable final placement inside or beside the target landmark.Plain textCopied 3. Sketch to simulationWith Argon, a developer can snap a photo of a rough pen-and-paper sketch—showing an Apollo 2 humanoid throwing a ball along a parabolic arc to topple a 3-2-1 stack of boxes on a table—and have Argon automatically construct the GPU-accelerated mujoco_warp scene, synthesize a balanced 76-DoF whole-body and five-finger throw trajectory, verify 6/6 target knockdown in closed-loop physics, and export a Blender 4.5 Cycles render.Sorry, your browser doesn't support playback for this videoSketch to Simulation — Hand-Drawn Sketch to Dexterous Humanoid Trajectory & Cycles Render.Prompt:[Attached: hand_drawn_throw_sketch.jpg]Translate this hand-drawn sketch into a physically verified humanoid simulation in `mujoco_warp` and a raytraced Blender Cycles video:1. Scene & Kinematic Setup from Sketch: - Parse the spatial layout from `hand_drawn_throw_sketch.jpg`: place the 76-DoF humanoid (32-DoF body equipped with two 22-DoF five-finger dexterous hands) at the origin facing a table holding a 3-2-1 pyramid of 6 rigid target boxes, with a projectile sphere grasped in the right five-finger hand.2. Whole-Body Balance & Dexterous Throw Synthesis: - Synthesize a 50 Hz whole-body throwing trajectory (140 steps, 2.80 s) across four phases: (1) stable stance & 5-finger power grasp closure, (2) torso/shoulder wind-up with ankle-hip Center-of-Mass compensation, (3) forward whipping acceleration, and (4) synchronized 22-DoF five-finger opening at peak hand velocity so the ball launches on a ballistic arc into the pyramid's second-tier center of mass. - Verify in `mujoco_warp` contact physics that the humanoid maintains upright balance (< 3 cm pelvis drift) and knocks down all 6 target boxes.Plain textCopied Unlocking physical understanding with Argon1. Hindsight labeling and data qualityArgon has precise physical understanding of dexterous manipulation and can provide detailed segmentation and text labeling from video. Argon is able to identify poor manipulation segments such as pauses, unstable grasps and fumbles. Given an overhead video of a bimanual ALOHA 2 robot folding a square sheet of paper into an origami fox over 136 seconds, Argon produces a time-aligned 4-level hindsight annotation hierarchy: Episode Goal (G1), 6 contiguous Semantic Subtasks (G2), 17 contact-level Atomic Primitives (G3), and automated Quality Control (QC) reweighting that holds task progress flat during fumbles and retries.Sorry, your browser doesn't support playback for this videoArgon Multi-Granularity Hindsight & Telemetry HUD.Prompt:[Attached: Aloha2_Origami_20250309.mp4]Produce a time-aligned multi-granularity hindsight annotation JSON with four levels:1. G1_episode_instruction: Concise imperative sentence describing the overall manipulation goal.2. G2_subtasks: Contiguous semantic subtask phases specifying which arm(s) anchor vs. fold/rotate the object.3. G3_atomic_primitives: Fine-grained contact-level primitives (3–15s) aligned with grasp closure, crease sweeps, release, or retry attempts.4. quality_control_and_reweighting: Hold task_progress_pct flat during non-progressing slip/delamination retry loops and downweight them (0.15x–0.25x) for Behavior Cloning.Plain textCopied 2. Find the odd ballTo recreate the classic 12-ball balance scale puzzle with robots, Argon builds a bimanual Franka FR3 MuJoCo environment with 12 visually identical pool balls (one secretly heavier or lighter) and an articulated Roberval two-pan balance scale. It then writes a vision-only closed-loop solver that triangulates the balls from stereo RGB cameras, visually reads the scale tilt after each weighing, and isolates the odd ball into the Heavy (Red) or Light (Blue) tray within 3 weighings.Sorry, your browser doesn't support playback for this videoFind the Odd Ball (Vision-Only Closed-Loop Solver): Dual-arm Franka FR3 weighing candidate balls on the Gold/Teal pans, reading stereo scale tilt , and isolating the confirmed Heavy odd ball into the Red tray in 3 weighings.Prompt:# Prompt 1 — Physics-Accurate MuJoCo Environment & Benchmark GenerationMake a MuJoCo environment where a dual-arm robot (two 7-DoF Franka FR3 arms with parallel-jaw grippers on a shared tabletop) is tasked to solve the classic 12-ball balance scale weight puzzle from https://www.mathsisfun.com/pool_balls_solution.html.1. Create 12 visually indistinguishable pool balls in a 2x6 starting rack (11 standard 150 g, 1 odd ball either 270 g or 30 g).2. Build a functional two-pan Roberval balance scale at the center of the table with parallelogram equality constraints (`mjEQ_JOINT`) so the Left (Gold) and Right (Teal) pans stay horizontal, plus Red (Heavy) and Blue (Light) submission trays.3. Track distinct settled weighings (`weighing_count`) and reward `1.00` for isolating the true odd ball in the correct Heavy/Light tray within