Accuracy and Efficiency Improvements for Fast Panoptic Segmentation

Important note: this project is not available anymore. It remains visible to show the type of student projects available at our group.

Description: During this project, you will improve an existing network for panoptic segmentation, to make it more accurate and efficient.

Type: Internship

Starting date: September 2021

Supervisor: Daan de Geus

General description:

Self-driving vehicles need situational awareness to be able to take appropriate actions. One way of gaining situational awareness is by applying scene understanding algorithms to cameras mounted on the vehicle. Panoptic segmentation is such a scene understanding task, which aims at recognizing all entities in a scene. Specifically, the goal is to predict, for each pixel, 1) a scene-class label (e.g., car, person, sky), 2) an instance id to distinguish between individual objects of countable classes (e.g., individual cars or persons).

Over the past years, several deep neural networks have been developed for panoptic segmentation, with varying accuracies and efficiencies. Especially for self-driving vehicles, where real-time applicability is key, efficiency of neural networks is very important. For this purpose, the Fast Panoptic Segmentation Network (FPSNet) was introduced [1]. FPSNet achieves efficient panoptic segmentation by making use of a fast object detection backbone, and an attention mechanism that indicates the location of individual objects.

However, recent work has outperformed FPSNet in terms of accuracy and efficiency, making use of various novel techniques. The goal of this internship is to leverage some of these techniques to improve the efficiency and accuracy of FPSNet.

Task description: During this project, you will implement and evaluate several improvements to FPSNet. These improvements include:

• More supervision of objects attended by attention masks.
• More accurate attention masks, instead of using bell-shaped blobs. These could be generated by letting the network learn borderness of objects.
• More advanced panoptic head architecture.
• More efficient and accurate detection backbone.
• Further optimization of hyperparameters and learning strategy.

Prerequisites:

• Theoretical knowledge about deep neural networks for computer vision.
• Experience with implementing a deep neural network for computer vision.

Interested? Send an email to d.c.d.geus@tue.nl, containing:

• Brief motivation letter
• List of relevant courses and grades

References

[1] D. de Geus, P. Meletis and G. Dubbelman, “Fast Panoptic Segmentation Network”, IEEE Robotics and Automation Letters, vol. 5, no. 2, pp. 1742-1749, 2020.