NVIDIA IsaacTeleop Translates XR Inputs into Robot Actions

NVIDIA has highlighted the capabilities of its IsaacTeleop framework, specifically focusing on the graph-based retargeting engine in version 1.4.145. By utilizing the "retargeters-lite" installation extra, which relies only on SciPy and NumPy, developers can run the entire pipeline on a standard CPU without needing a physical virtual reality headset or simulator. The core framework relies on a strict type system featuring TensorGroup and TensorGroupType to validate inputs, such as the 26 OpenXR hand joints in HandInput or the 14 data slots in ControllerInput, preventing data errors at write time.

Practitioners can build custom nodes, like a PinchRetargeter that measures thumb-to-index distance with a configurable threshold between 0.5 and 10.0 centimeters. IsaacTeleop also includes built-in components like the GripperRetargeter, which converts pinch distances into gripper commands with hysteresis, closing below 3 centimeters and opening above 5 centimeters. For spatial tracking, the Se3AbsRetargeter maps controller poses to a seven-dimensional end-effector target, while the Se3RelRetargeter calculates relative motion deltas using a scale factor of 10.0 and an exponential moving average alpha of 0.5.

To deploy these tools, developers construct a directed acyclic graph where leaf nodes feed into retargeters. An ExecutionCache ensures that shared inputs, like a controller pose, are computed only once per step even when wired to multiple downstream nodes. Before sending commands to a simulator, a ControllerTransform node applies a four-by-four homogeneous matrix to convert coordinates from the headset's anchor frame to the robot's world frame. A TensorReorderer then flattens the spatial poses and gripper scalars into a single action vector.

The framework also manages system states and complex hardware. The DefaultTeleopStateManager handles transitions between stopped, paused, and running states, while the LocomotionRootCmdRetargeter manages robot base movement with a default hip height of 0.72 meters. For complex manipulation, the TriHandMotionControllerRetargeter maps standard controller inputs to seven dexterous joint angles without requiring complex optimization libraries. Finally, parameter states can be saved to JSON files, allowing custom calibrations to persist across system restarts.