Sim2Real Robotized Object Manipulation

Basic Information

Students: Benjamin Piironen, Matias Mäki-Leppilampi, Pyry Aho

Project manager: Pyry Aho

Instructor: Almas Shintemirov

Other advisors: Tran Nguyen Le

Starting date: 10.1.2023

Completion date: 6.6.2023

Abstract

The purpose of this project was to investigate the viability and potential of the NVIDIA Omniverse Isaac Sim application in controlling a real robot arm manipulator through simulation. Another important goal was to gain a deeper understanding about the more general concept of “Sim2Real”, which involves bringing desired aspects of a simulation environment to reality. The focus of the project was on the relatively newly released Isaac Sim due to its unique features, high degree of realism, and its potential for use in future research. The scope of the project was limited to the Isaac Sim and Gazebo simulation environments, as well as relatively simple pick-and-place tasks, primarily due to time constraints and the efficacy of even simple simulation tasks in serving as a proof-of-concept for more advanced applications.

A simple pick-and-place scenario involving a simulated Franka Panda robot arm was first implemented using Gazebo and the MoveIt motion planning framework. This same scenario was then reproduced in Isaac Sim first using only the built-in controllers and communication tools included in Isaac Sim. An implementation utilizing a combination of Isaac Sim and MoveIt, which was in practice very similar to the previously accomplished Gazebo implementation, was also successfully carried out.

Another pick-and-place task involving stacking cubes of differing colors from various locations onto a specified point was also implemented. This involved using an object recognition library to process camera data transmitted through ROS to detect each cube, after which the robot arm would generate its motion trajectories to accomplish its given task. This more complex task was also successfully tested using a real Franka robot arm, serving as a promising example of using Isaac Sim for real robot applications in the future. With this, all of the primary goals of the project were successfully achieved.