Generating realistic digital twins and learning from them

Music City Center 

Digital twins are becoming increasingly important due to their ability to enhance operational efficiency while reducing costs. In the first part of my talk, I will focus on how we can generate digital twins of fractures in the camera or the camera enclosure itself and then overlay them on existing open-source datasets using realistic visualization. We compare our generated results to real broken glass images and show that they are distributionally similar. We also provide the adversarial effect of such generated images, when overlaid on existing datasets, paving a way to provide natural perturbations on images and learning from them.
In the second part of my talk, I will pivot towards using digital twin data in supervised object detection algorithms. The labeling operation required for creating ground truth data makes it a very labor intensive process. Alternatively, we focus on sim-to-real transfer - learning from simulation, and transferring onto a real environment. We perform robust object detection on a collected dataset, without having to label any instances in the target dataset. We show the results from the process, including the robustness of such object detection approaches as compared to learning from labeled examples.