Smart endoscopy – AI based polyp detection during the colonoscopy

Using deep neural networks for automated diagnosis


The aim of the Deep Colonoscopy project is to develop an automated system for detecting, classifying and documenting polyps and lesions in real time. To this end, the project is evaluating and training suitable deep neural network architectures that provide highly specific and highly sensitive data analysis while running on low-complexity hardware components.



Intelligentes Endoskopiesystem zur KI-basierten Polypendetektion
© Fraunhofer IIS
Mapping of stomach and intestinal walls during endoscopic examination using AI
Vorklassifizierung von Polypen nach der Paris-Klassifikation
© Fraunhofer IIS
Pre-classification of polyps according to the PARIS system using smart endoscopy

Designed to support physicians in their diagnosis, this intelligent endoscopy system

  • creates a real 3D model of the stomach and intestinal walls in real time; the panoramic images generated provide the greatest orientation for mapping polyps directly,
  • highlights suspicious areas of the image using markings such as coloring,
  • automatically detects polyps and other lesions (adenomas, etc.) and pre-classifies them according to the PARIS system,
  • automatically generates documentation of the detected lesions and additional information (size, location, surface structure, etc.), including pre-filled text fields.


Colorectal cancer is one of the most common types of cancer. Colonoscopy is considered the most reliable method for detecting colorectal cancer at an early stage, especially for detecting adenomas, which are its precursor.

Physicians face the challenge of detecting, differentiating and documenting any lesions (in particular small polyps, flat neoplasms, bleeds, etc.) – often under time pressure. During the examination, physicians must rely solely on their prior knowledge and experience as they move the tip of the endoscope along intestinal walls one section at a time. They have no access to a 3D image of the intestine or pre-classification in real time. Their diagnosis is documented only after the examination has taken place.

Against this backdrop, the project consortium aims to provide physicians with a real-time support system in order to minimize diagnostic risks and reduce the time taken for the diagnosis.


Smart endoscopy system with AI-based diagnosis supports physicians

  • AI-based video analysis for identifying and classifying polyps ensures a higher rate of detection, enhancing levels of patient safety
  • Automatic generation of the diagnosis documentation saves time and provides greater quality assurance
  • Structured diagnostics and compatible interfaces enable integration irrespective of manufacturer
  • White light endoscopy keeps costs low





  • E&L medical systems, Erlangen (consortium lead partner)
    • Development of software tools as an add-on to the Clinic WinData (CWD) image and diagnostic documentation system
    • Provision of training and test data, annotation of the image data
    • Connection to a suitable endoscopy system
  • Malteser Waldkrankenhaus St. Marien, Erlangen
    • Creation of the function demonstrator
    • Scientific evaluation using phantoms, anonymized endoscopic image sequences
  • Nexus AG, Munich branch
    • NEXUS / NBBNG diagnostic platform Research of suitable data formats and communication protocols for later integration into HIS
    • Concept development and evaluation of HIS communication
  • Fraunhofer IIS, Erlangen
    • Design, development and establishment of a combined hardware and software framework / suitable infrastructure (system, memory, graphics cards, control software) for training, validating and testing deep neural networks
    • Research, evaluation, selection and implementation of suitable architectures
    • Training, validation and testing of the deep neural networks