< Back to Main Page

[Archived page]

Workshop on Closing the Reality Gap in Sim2real Transfer for Robotic Manipulation

Full Day Workshop at R:SS 2019 (Freiburg, Germany) - Building 101, Room 00 026 - June 23, 2019

NEWS: Camera-ready of accepted papers & speakers slides are online

Overview · Invited Speakers · Topics · Call for Contributions · Important Dates · Accepted Papers · Schedule · Venue · Organizers

Physical simulation is an important tool for robotic manipulation. Although simulation has been well-established for robotics education and integrated robot software testing, there is an ongoing debate about transferring manipulation capabilities learned in simulation to reality, a concept termed sim2real transfer.

In this workshop, we will shed light on the following questions: What is the potential impact of sim2real transfer on robotic manipulation? What methods exist and work best in the context of manipulation? To what extent can sim2real transfer reduce or avoid training on real robots altogether?

Simulation draws its appeal from the fact that it is much faster, cheaper, safer and more informative (e.g., auto-generated labels) than real-world experimentation. Recent advances have shown how to take advantage of simulation and address the sim2real transfer for tasks such as object detection, autonomous driving and grasp point detection. However, sim2real transfer for general manipulation skills still raises significant challenges:

  1. Contact simulation: Manipulation is about contact. However, simulating physics and in particular contact is a highly complex problem. The reason is that contact formation depends on a wide variety of physical properties such as friction, object deformability, etc. Since learning-based methods exploit any circumstance that facilitates finding a solution, they are prone to overfit to imprecise modeling and thus fail to learn transferable policies.
  2. Simulating closed-loop manipulation: Manipulation is not a single-shot but a temporal process. Therefore, even slight errors at the beginning of a simulation sequence tend to accumulate and then lead to simulating unrealistic behavior and learning non-transferable policies.
  3. Curse of dimensionality: Many state-of-the-art approaches to sim2real rely on sampling a large number of policies and environmental conditions. However, sampling becomes exponentially more difficult with the number of task dimensions. This raises the question whether current techniques will scale to the high-dimensionality of manipulation problems, e.g. involving high-DOF manipulators.
  4. Sensor fidelity: Certain sensor modalities are much more difficult to simulate than others, e.g. simulating RGB or force/torque is more challenging than depth. Despite the recent advances in image synthesis and domain transfer techniques, we still do not fully understand what statistical properties of sensors need to be simulated in order to successfully learn from them.

Invited Speakers [Top]

Topics [Top]

Call for Contributions [Top]

Participants are invited to submit extended abstracts (maximum 2 pages in length, excluding references) related to the aforementioned topics.

Accepted abstracts will receive a poster presentation slot and, if equipment is available, a video presentation slot during the poster session.

Submission website: https://cmt3.research.microsoft.com/SIMREALRM2019

Important dates [Top]

Accepted Papers [Top]

Schedule [Top]

The most up-to-date schedule can be found on our Google calendar: [View] [Subscribe]

09:00 - 09:15 Welcome and introduction  
  Session 1: Sim2real for Manipulation I  
09:15 - 09:40 Kris Hauser: Real2sim: generalizable learning to simulate complex contact phenomena [slides]  
09:40 - 10:05 Gilwoo Lee: Bayes-optimal reinforcement learning for sim2real adaptation  
10:05 - 10:30 Chelsea Finn: Learning and Adapting in Diverse, Dynamic Environments [slides]  
10:30 - 10:45 Poster Teasers  
10:45 - 11:30 Posters Session I / Coffee Break  
  Session 2: Sim2real for Grasping  
11:35 - 12:00 Edward Johns: Three Benefits of Simulators for Zero-Shot Transfer [slides]  
12:00 - 12:25 Josh Tobin: Beyond Domain Randomization. [slides]  
12:25 - 12:45 Yunfei Bai: Learning to grasp using simulation and deep learning [slides]  
12:45 - 13:30 Lunch break  
  Session 3: Sim2real for Locomotion  
13:35 - 14:00 Erwin Coumans: Sim-to-real for quadruped locomotion [slides]  
14:00 - 14:25 Jean-Baptiste Mouret: Data-efficient Adaptation to Damage is a Reality Gap Problem [slides]  
14:30 - 15:15 Posters Session II / Coffee Break  
  Session 4: Sim2real for Manipulation I  
15:20 - 15:45 Fereshteh Sadeghi: Domain Invariant Semantic Robot Navigation  
15:45 - 16:10 Emo Todorov: Robustness of model-based control [slides]  
16:10 - 16:35 Russ Tedrake: Robust full-stack manipulation (perception, planning,
and control) via simulation-based design and verification [slides]		  
16:45 - 17:30 Speaker Panel Discussion  

Venue [Top]

The workshop takes place in Building 101, Room 00 026.

The poster sessions takes place in Building 101, Room 00 019.

Please check out the RSS website and Google maps for more detailed information on the workshop’s location.

Organizers [Top]

Sebastian Höfer Sebastian Höfer is an applied scientist at Amazon Research - Robotics AI headed by Siddhartha Srinivasa. Before joining Amazon, he received his Ph.D. with Oliver Brock at the Robotics & Biology Lab, Technische Universität Berlin.
Ankur Handa Ankur Handa is a senior research scientist at the NVIDIA robotics lab run by Dieter Fox. Prior to that he was a research scientist at OpenAI. He received his Ph.D. with Dr. Andrew Davison and spent two years at University of Cambridge in Prof. Roberto Cipolla’s lab as a post-doctoral researcher.
Kamal Kuzhinjedathu Kamal Kuzhinjedathu is a senior applied scientist at Amazon Research - Robotics AI headed by Siddhartha Srinivasa. Before joining Amazon, he was a principal engineer at Microsoft working on perception for Microsoft Hololens.
Marc Toussaint Marc Toussaint is professor for Machine Learning and Robotics at the University of Stuttgart since 2012 and Max Planck Fellow at the MPI for Intelligent Systems since November 2018. His research aims to bridge between machine learning, control theory and AI planning, motivated by fundamental questions in robotics. Reoccurring themes in his research are appropriate representations and priors to enable efficient learning, reasoning and manipulation in the real world, combining geometry, logic and probabilities in learning and reasoning, and active learning and exploration.
Dieter Fox Dieter Fox is a Professor in the Department of Computer Science & Engineering at the University of Washington. He received his Ph.D. in 1998 from the Computer Science Department at the University of Bonn. He joined the UW faculty in the fall of 2000. He is currently on partial leave from UW and joined NVIDIA to start a Robotics Research Lab in Seattle.