If autonomous vehicles are to make the correct decision in every conceivable situation, they must be able not only to locate their own position in traffic, but also to reliably capture their environment with precision. To do this, vehicles must have the ability to collect and fuse data from sources such as laser, camera, and radar sensors. For the algorithms that process such sensor data, artificial neural networks have become indispensable tools. However, these networks need fast, efficient, flexible hardware, which is precisely what the “KI-FLEX” project has been successfully researching over the past four years. “This is an important step toward the safe mobility of the future,” says Michael Rothe, who heads the Embedded AI group at Fraunhofer IIS.

On top of every traffic situation

Systems must be able to unambiguously detect and identify objects and road users in traffic situations. As such, the importance and utility of the individual sensors vary accordingly. Both the traffic situation and the weather and light conditions have to be taken into account in order to ensure safe autonomous driving. Moreover, the systems must be able to respond flexibly to potential sensor failures or adversarial attacks in their data. To address this requirement, the project partners developed resource-optimized approaches for the early and late fusion of camera data, lidar data, and detected objects along with an AI-based monitoring system. These components allow vehicles to reliably respond to changed situations by adjusting the algorithms used.

Reconfigurable AI system

Artificial neural networks are currently developing at a rapid rate. The growing number of architectures is making increasing demands on the hardware and software. For this reason, “KI-FLEX” employs a heterogeneous hardware architecture made up of FPGA and ASIC AI accelerators in order to implement the neural networks for object detection in camera and lidar data. This flexibly reconfigurable and programmable AI accelerator system anticipates the future to some extent, as the hardware will be able to support emerging neural network designs. Furthermore, the hardware platform’s computing resources can be allocated dynamically according to load.

The AI chip developed in the project also offers considerable advantages with regard to power consumption, processing speed, and cost savings compared to conventional multi-purpose processors (CPUs) or graphics processing units (GPUs).

Germany-wide research initiative

The project “KI-FLEX – Reconfigurable hardware platform for AI-based sensor data processing for autonomous driving,” which launched in September 2019, was funded by the German Federal Ministry of Education and Research (BMBF) within the guidelines on promoting research initiatives in the field of “AI-based electronic solutions for safe autonomous driving (AI element: autonomous driving).”

Led by Fraunhofer IIS, the project consortium comprises several German research and industry partners: Infineon Technologies AG, videantis GmbH, the Technical University of Munich (Chair of Robotics, Artificial Intelligence and Real-Time Systems), the Fraunhofer Institute for Open Communication Systems FOKUS, the Daimler Center for Automotive IT Innovations (DCAITI, Technical University of Berlin) and the Friedrich-Alexander-Universität Erlangen-Nürnberg (Chair of Computer Science 3: Computer Architecture).